まずM5Stack、M5Stack用電池モジュール、LEPTON3 or 3.5、LEPTONブレイクアウトボードを手に入れてください。 以下必要部品の入手先とLEPTON画像再現までの手順です。
また、LEPTON3.0と3.5は自動判別します。
This application requires installation of the following separate parts and source. Please procure parts and install software according to the order.
First, get the M5Stack,M5Stack battry module ,development board,LEPTON 3 or 3.5, LEPTON breakout board.Below is the procedure from where to obtain the required parts to LEPTON image reproduction.LEPTON3.0 and 3.5 are automatically identified.
1. M5StackとM5Stack電池モジュールを用意してください。Prepare the M5Stack and M5Stack battry module.
2. LEPTON3.0又は3.5を用意してください。Please prepare LEPTON3.0 or 3.5
The LEPTON sensor itself is completely waterproof, and it is quite strong for impact, so it does not bother with adverse conditions, but detachable shutter parts have a breakable disadvantage. Please handle with care. LEPTON regularly closes the shutter and calibrates the image, but if the shutter breaks, afterimages will remain, such as image failure will come out. Currently it seems that we do not sell replacement parts for the shutter, so careful handling is necessary.
3. BREAKOUT BOARDが必要です。 BREAKOUT BOARD is required.
When initializing iPhone as usual, ssid becomes Japanese. Please correct it to half-width alphabetical as it may cause an error.
Choose Settings -> General -> Information -> Name -> Current iPhone Name> to change the name. Changing the name does not affect the behavior of the iPhone itself.
8. M5Stack Arduino ソースをダウンロードします。Download the M5Stack Arduino source.
Please download from the above URL.Open ArduinoIDE and double click on the file you downloaded now. Can you put it in the folder? As you are asked, choose YES. We commented on the stabilization of OSC and SPI, the elimination of UDP packet loss problem, and the solution method of WSP reset specific to ESP.
Set the name of iPhone to ABiPhone, set to 11111111 -> Mobile data communication -> Enter the password by checking the Wi - Fi password.
When GPS is turned on, it glows shimmerly periodically. I think that you can see that there is not much effect on the picture although it surely loses the packet. When BlueTooth is turned on, it glows a lot and the frame rate drops.
11. M5StackとLEPTON BREAKOUT BOARD を接続します。Connect M5Stack and LEPTON BREAKOUT BOARD.
12. WROOM02(ESP8266)ボードにプログラムを転送します。Transfer the program to WROOM02 (ESP8266) board.
When compilation and transfer are done, it completes. Although occasionally an error may appear, please transfer again in that case. (USB is not recognized in many cases)
13. いよいよサーモグラフィー開始 The thermography finally starts !!
iPhoneX、11Proでの動作確認をしています。iPadでも動くと思いますが、両者ともCPUはA9以上が必要です。iPhone 11 Pro MAX、Xs、Xs Pro、XR、8、8Plus、他のiPadに関してはシュミレーターで動作を確認・調整しています。
I am checking the operation with X,11Pro. I think that it will work on general iPad, but both require A9 or more CPU. For iPhone Xs,Xs Pro,XR,8, 8 Plus, and other iPads, the operation is checked and adjusted with a simulator.
It is not easy from installation to first confirmation of operation, but for everyone who is accustomed to Arduino it is a completely problem free process. While keeping security, it moves with 3 steps (LEPTON side switch ON -> Internet share -> iPhone application launch) after the second time. No troublesome WiFi selection, IP, password entry etc are required at all. Also, even if you use tethering, you will not communicate with the outside, so there will be no communication fee for Thermal Cam 3 operation.
Thermal Cam for BOSONは、安価で小型のRaspbery PI Zero をFLIR BOSON に接続し、Wi-FiテザリングI経由でiPhoneに画像を再現できます。動画スピードは、10Mhz(10fps)以上のBOSONでは最高30fps前後です。
PI ZeroとiPhoneはWi-Fiで相互に交信し合います。PI側ではカメラの持つ性能を最大に発揮させるため8bit処理したRAW16グレースケール画像をiPhoneへ送っています。制御命令はiPhoneから送信しているので、市販のBOSON単体とコンピュータをつないだだけでは実現できない暫定温度測定・感度の微調整・自由な色づけが可能になりました。さらに画像データはmjpegなどで圧縮していないためBOSON本来のノイズの少ないくっきりした画像が再現されます。iPhone側では画像を特性に合った方法で拡大しているので、緻密な画像表現が可能です。市販されているBosonは温度測定機能を持っていませんが、後述したキャリブレーションを行うことで、ある程度正確な温度測定が可能になりました。
PI Zeroは他のPIシリーズに比べてスピードや起動が遅いことから、OSのやソースのインストールなど非常に時間がかかります。その代わり他のPIでは実現できない携帯性が加わることになります。RaspiOSを含めて全体のインストールは慣れていても1日くらいの作業量です。途中でエラーが出た場合は最初に戻って再度挑戦して下さい。
We add object detection forthermography. One of feature for Thermal Cam is night vision and our system is divided display from FILR sensor.
ESP8266 helps to transmit WiFi longer and light weight low battery consumption all at the same time.
Furthermore, various applications are possible by adding image recognition AI.
However, AI function cannot be executed just by downloading Thermal Cam application. Please download the Pre-learned file first referring to the following.
ESP8266 helps to transmit WiFi longer and light weight low battery consumption all at the same time.
Furthermore, various applications are possible by adding image recognition AI.
However, AI function cannot be executed just by downloading Thermal Cam3x application. Please download the Pre-learned file first referring to the following.
特徴は? New feature ?
iPhone Xs,Xs Maxm,Xr,11,11 Pro,11Pro Max,iPad Pro 11,12.9inchに関して (A12,A13 Bionicを使っているiPhone,iPad)、「設定」YOLO ONトグルスイッチをONにすることで、画像認識機能が働きます。*1
For iPhone Xs, Xs Max, 11, 11 Pro, and 11 Pro Max, the image recognition function works by turning YOLO ON toggle switch at the setting of iPhone.
* 1 The recognition engine uses YOLOv3. Since this is not a Tiny version, it demonstrates excellent recognition ability.Pre-learned files can be freely selected, and the weight data prepared by the user.
* 2 Using the recognition results, the user can freely set Siri voice in English or Japanese, alarm, camera flash and display number of objects recognized, time stamp.
* 3 You can freely change the above alert and the text of the target object (hereinafter referred to as Classification). This requires a general text file edit software. Although it is a paid application, it is recommended to install “Textor” or "Edidown" in advance.
To perform image recognition, you must first download the Pre-learned data. The Pre-learned file will be 100Mbyte ~ 250Mbyte, so to start with, please use one of them after checking your own iPhone capacity.
The following three types of the Pre-learned models is available at the moment.
(Standard RGB image is targeted: 80 kinds of object detection, test data to confirm how much is recognized by infrared image. This is a general COCO 16bit)
(Standard RGB image is targeted: 80 kinds of object detection, test data to confirm how much is recognized by infrared image. This is a general COCO 32bit)
The download method is explained below, but please specify the URL and unzip the downloaded zip file, and then put it in the folder dedicated to Thermal Cam3x.
Change the safari download location to your iPhone internal file. Although iCloud is the standard download location, we recommend to change to iCloud after getting used of it.
1.2 上記のアドレスの中から学習データを選択します。Select Pre-learning data from the above addresses.
Rock✊-Paper✋-Scissors✌️ of your hand or your face😃 can be detected, good enough to check feature to start with. You can see how to judge the shape of your hand and our model can detect at any angle of your hand. As a feature of the infrared camera, even if a thermal image of a car captured like Web Camera, the target cannot be easily determined. It must realy generate heat. In other words, if it is in the shape of a hand, it can be judged immediately at home, and in order to judge the car, you must go outside and capture the actual cars.
1.3 ファイルアプリを開き以下の操作を実行します。Open the file app and do the following:
After unzipping, the zip file is no longer needed, so delete it and move the unzipped folder to a folder dedicated to the app (Thermal Cam3x or Thermal Cam B folder).
Thermal Cam3x 又は Thermal Cam B のフォルダを選択し、w_tフォルダを移動します。
Select the Thermal Cam3x folder and move the w_t folder.
All downloaded folder names will be w_t. You can check one folder and three text files as shown below. Click the w_t folder to check the contents of files such as README. You can also check what can be recognized in the w_t.names file. Although it is possible to extend by editing the text file, editing work seems to be impossible with the iPhone standard application, so it is necessary to work with text file editing application “Textor” etc.
Other text file editing apps
Edidown(Free app:I recommend this now)
Pythonista (paid app:using editing function)
* Thor is not currently available on the app store. GitHub-louisdh / textor: Must be downloaded from A plain text editor for iOS and compiled.
The file contents are as follows. Please note that the extension(.txt & .names) may not be confirmed on iPhone.
w_t.mlmodelc Folder containing Pre-learned files.
Never change the contents.
w_t.names
The name of the class is written across the line feed code(/n). The class name of the bounding box is displayed using this name. You can change the name, but please be careful not to change in Japnese(must not be 2bites language) andthe total number of lines. Pre-made image recognition models have a limited number of recognitions.
w_t(.txt)
This file sets alerts after recognition and the overall recognition rate (threshold). By changing the parameters in this file, you can change alerts related to recognition results in various ways.
Please confirm “3. Setting method of recognition result alert” for contents and change method.
README.txt
Contains the characteristics of the downloaded learned file.
When selected in Thermal Cam settings, Thermal Cam3x image recognition starts.
Normally, when you use the GPU(A13 Bionic) fully, your mobile device gets a lot of heat, but this app does not get extremely hot because it controls the speed. However, avoid walk with running the app in a pocket. In addition, caution as the battery is consumed quickly. When using for a long time, ensure the power supply and use a cooling fan for iPhone.
The figure below is an image of w_t.txt opened on the “Textor” editing screen. This section explains each setting. Do not change the characters on the left-hand side including the =. In addition, it is necessary to enter a numerical value or a character on the right side of =. Please note that the standard value will be applied if a value outside the specified range is entered.
Set the threshold for the overall recognition rate (all classes that can be recognized). This value ranges from 0.1 to 0.9. Decreasing the number increases misrecognition. The standard is 0.4.
iou_all = 0.3
Specifies the overlap factor of the bounding box. This value ranges from 0.1 to 0.9. Increasing this number will increase the number of overlapping bounding boxes with the same class name and make the image annoying. The standard value is 0.3.
By adjusting these values and adjusting Dynamic Range !! slider, you can focus on the target class. It may also need to be changed slightly depending on the Pre-learning model.
The following settings issue a special alert based on the recognition result for the specified class. Currently there is only one target class.
class = 7
Specify the number of classes of Pre-learned files or 0. The target class number. Specify 1 as the top of the class name written in w_t.names. Enter 0 if no alert is required. In the case shown below, class 7 refers to “Goo”.
Enter the target class threshold. When 0.7 is set, it means that alert processing is executed when the target class number 7 has a recognition rate of 70% or more. Specify in the range of 0.1 to 0.99. The standard value is 0.7.
disply = 1 : Class name etc. are displayed at the lower left of the screen.
0: Class name is not displayed at the bottom left of the screen.
2: Change the numerical part of the bounding box to the time stamp (standard is the recognition rate display). This setting does not depend on the specified class.
この設定例でclass番号を3に変更した場合、例えばLEPTON端末を自転車の後ろに設置して、iPhoneアプリを実行させ、後ろから70%以上の認識率の車が迫ってくると Please note that is a car !! とiPhoneが教えてくれます。日本語指定の場合は「車が来ます」でも良いですね。赤外線カメラですから当然真夜中でも!!
country = us
Specify the type of alert.(All specified with two single-byte characters)
ja or us: If the recognition rate is higher than the specified rate, the text written in the Siri voice below will be read out in us=English or ja=Japanese.
so: Abbreviation for “sound”. If the specified recognition rate is exceeded, a warning sound is emitted. Only “whistling” sounds.
fi: Abbreviation for flash. If the specified recognition rate is exceeded, the camera flash will fire twice.
voice = Plaese_note_that_is_a_car
Enter the text to be uttered when country is ja or us. For English, enter up to 60 characters, for Japanese(2bites), up to 30 characters. it should be sound within 1.5 seconds.
For English, be sure to replace the blank with an underscore(_). If it is set to ja and English is used, English utterance will be non-native.
If you change the class number to 3 in this setting example, for example, if you install the LEPTON terminal behind the bicycle and run the iPhone application, if a car with a recognition rate of 70% or more approaches from the back, Plaese note that is a car !! and iPhone(Siri) will tell you. In the case of ja, “car will come” is also in Japanese. Of course, it’s an infrared camera, even at midnight ! !
Actually tested … The advantage of the alarm effect was definitely experienced on the narrow roads that are particularly dangerous for bicycles. Actually, the alarm is sounding at the recognition rate of 70% displayed as Car in the lower left. (This is a video at the development stage.)
When you edit the contents of w_t.txt with “textor” and save it with Done, the settings will be reflected immediately, so find the best alert while changing your setting.
From here, it is for those who have sufficient Deep learning experience and knowledge, and who can use an environment where Linux machines with NVIDIA GPUs can run. The first involves a very complex installation(according to https://github.com/AlexeyAB/darknet), but not much after the second time. The outline is as follows. When you understand the contents, you can improve the efficiency by grouping a series of tasks on the Mac side into Python.
Learning on linux machine-> Copy learned weights file to mac-> Convert to h5 file-> Convert to coreml file-> Convert to coremlc file-> Create three txt files-> Compress to zip
4.1 録画した赤外線画像データをアノテーションする。Annotating LEPTON images and learning
In order to run on iOS, the following settings are required.
Use yolov3-voc.cfg or yolov3.cfg as the configuration file (CFG file).
The size of the input image is limited to with = 416 heights = 416.
The configuration file of YOLOv3.cfg is also explained on the reference page, but it is necessary to change a total of 10 or more locations in relation to the class
Download and use darknet53.cov.74 as the initial weights file to speed up learning convergence.
Create Anaconda environment for python3.7 below and install YAD2K and keras. Each installation method is described at the beginning. However, when installing Keras for the first time, only the Tensorflow installation command is used. Conda install -canaconda tensorflow == 1.14 That is, tensorflow version should be 1.14.
After entering the keras-yolo3-master folder, the following work is also performed in the anaconda environment.
First of all, convert.py is executed, but an error will occur if this is done, so change the 88th line of convert.py to input_layer = Input (shape = (416, 416, 3)).
Next, put the corresponding weights file in the same directory and the cfg file used for learning.
Change the parameter set at the beginning of the cfg file to bach = 1 subdivisions = 1, and create a new h5 file with the following command. Change the cfg and weight file names to the copied file names, and give the .h5 file a name of your choice.
In this example, an h5 file is created in the data folder.
w_t.names : The class name file used for learning is used by changing the file name.
w_t.txt: Refer to “3. Setting method of recognition result alert”.
REDME.txt : Name and content are optional and are not actually used. All work files must have the same name, so please enter a description of the data so that you can check it later.
4.8 合計4つのファイルを新たに作成したw_tフォルダに移動する Move a total of 4 files to the newly created w_t folder
Compress the w_t folder to zip. The easiest download location for iPhone is iCloud. Since this Thermal Cam3x folder name is limited to w_t, please manage folders for each model.
Thermal Cam3x transmits and receives data using OSC, and can communicate with synthesizers using OSC communication, multiple ESP8266s, and computers. This version also supports AI image recognition that allows users to freely use image recognition data prepared by the user, so various ideas using infrared cameras can be realized on the iPhone or iPad. In the future, we will continue to strengthen awareness alerts, but I hope that your little ideas will be produced with fun.
*1:残念ながら内臓GPU性能に依存するためiPhone X以前の機種では使用できません。また事前に学習済みファイルを所定のフォルダにダウンロードする必要があります。Unfortunately, it depends on the built-in GPU(Bionic chip) performance and cannot be used with iPhone X or earlier models. In addition, it is necessary to download the learned files to a predetermined folder in advance.
*2:現在公開しているテスト用学習済みファイルは3種類です。There are three types of test-learned files currently available.
*3:複数のESP8266にメッセージを送るなど今後機能を増やしていく予定です。We plan to add more functions in the future, such as sending messages to multiple ESP8266.
過去2回Maker Faire Tokyoに出展経験がありますが、会場ではWIFIが繋がりにくく、急遽ビデオによる紹介しかできずに悔しい思いをしていました。そのため、今回はESP8266の代わりに有線LANでもWIFIでも対応できるラズベリーパイ3(以降PI3)でLEPTON3を動かすことにしました。海外でこの組み合わせに成功した事例が公開されているので楽勝かと思ったのですが、大きな問題がありました。Linuxのマルチタスク処理が主な原因で、LEPTON3+PI3では数分〜数時間で止まってしまうのです。FLIRがこの組み合わせのソースを公開していない理由が、体験を通して理解させられることになりました。でも、私たちは基盤とソフトの工夫で、これを克服することができています。PI3単独で赤外線画像(暗視画像)を表示するだけでなく、iPhoneの代わりに使ったNvidiaのTX2やXavierからでもLAN経由で画像制御と動画表示が可能です。まずこれが1つ目の展示です。