Sony Released the World's First Built-in Intelligent Vision Sensor

Intelligent Vision Sensor

The world's first image sensor with built-in AI comes out. This is the heavy news from Sony recently:

Announced that two intelligent vision sensors will be released soon that is IMX500 and IMX501. They integrate a logical processor and memory, so each module can work as a microcomputer. At present, this sensor can capture still images with 12 million pixels and supports 4K @ 60fps video shooting. The shipment time of IMX500 and IMX501 is April and June this year, respectively, and the after-tax prices are 10,000 yen (approximately 93.02 US dollars) and 20,000 yen (approximately US dollars).

The new sensor has the following characteristics:

1. The stacked structure consisting of a pixel chip and a logic chip is equipped with an image sensor with AI image analysis and processing functions on the logic chip.
2. The signal acquired by the pixel chip is processed by the AI ​​on the sensor without the need for a high-performance processor or external storage, which can help the development of the edge AI system. The sensor outputs metadata (semantic information belonging to image data) instead of image information, thereby reducing the amount of data and ensuring privacy and security.
3. In addition, AI can provide a variety of functions for a wide range of applications, such as real-time target tracking under high-speed AI processing. You can also reprogram the memory using the location according to the user's needs or system to choose different AI models.

• 

According to Sony, the newly released image sensor has greatly expanded the research and development space of AI cameras, and is expected to be widely used in retail and industrial equipment industries, and will help to build optimized systems connected to the cloud.


• 
  
  
  Main functions of the first built-in AI image sensor
  Next are the four main functions of IMX500 and IMX501. Image sensor with AI processing function The pixel chip is back-illuminated, with about 12.3 million effective pixels, which can capture image information from a wide-angle of view. In addition to the traditional image sensor operation circuit, its logic chip is also equipped with Sony's original DSP, which is specially used for AI signal processing and storage for AI models. This configuration does not require high-performance processors or external storage, making it ideal for edge AI systems.
• 
  
• Metadata output
• The signal collected by the pixel chip is processed by the ISP, and the AI ​​operation is performed at the processing layer of the logic chip, and the extracted information is output as metadata, which can reduce the amount of data that needs to be processed. At the same time, not outputting image information helps reduce security risks and ensure privacy.

• 
• 
  
  In addition to the images captured by traditional image sensors, users can also choose data output formats according to their needs and uses, including ISP format output images (YUV / RGB) and ROI specific area extraction images.
  
  
  High-speed AI processing
  When recording video using a traditional image sensor, each frame of video data needs to be sent for AI processing, resulting in a large amount of data transmission, making it difficult to present the results in real-time.
• 
  

• 
  
Sony's new sensor products perform ISP processing and high-speed AI operation on the logic chip (processing MobileNet V1 takes only 3.1 milliseconds), completing the entire process in one video frame. This design can ensure high-precision, real-time target tracking while recording video. Free choice of AI model. Users can write the AI ​​model of their choice into the embedded memory, and rewrite and update it according to their needs or system usage.
• 
  

• 
  For example, when multiple cameras equipped with new sensors are installed in retail locations, a single type of camera can be used in different locations, environments, times, or purposes: When installed at the entrance of the venue, it can be used to count the number of visitors; When installed on store shelves, it can be used to detect shortages in inventory; When installed on the ceiling, it can be used to draw a heat map of the shop's passenger flow. In addition, the AI ​​model in the specified camera can be rewritten from a model for detecting heat maps to a model for identifying consumer behavior.
  
  
  Finally, I provide the key parameters of IMX500 and IMX501.

• Model name		IMX500 (bare chip product)	IMX501 (package product)
  Number of effective pixels		4056(H)×3040(V), approx. 12.3 megapixels
  Image size		Diagonal 7.857 mm (1/2.3 type)
  Unit cell size		1.55μm(H)×1.55μm(V)
  Frame rate	Full pixel	60fps
  	Video	4K（4056×2288）60fps
  		1080p 240fps
  	Full/video+AI processing	30fps
  	Metadata output	30fps
  Sensitivity (F5.6 standard value)		Approx. 250LSB
  Sensor saturation signal level (minimum value)		Approx. 9610e-
  Power supply	Analog	2.7V
  	Digital	0.84V
  	Interface	1.8V
  Main functions		AI processing function, ISP, HDR shooting
  Output		MIPI D-PHY 1.2 (4lane)/SPI
  Color filter array		Bayer array
  Output format		Image (Bayer RAW), ISP output (YUV/RGB), ROI, metadata
  Package		-	Ceramic LGA 12.5mm(H)×15.0mm(V)

• 
  
• Reference
• https://www.sony.net/SonyInfo/News/Press/202005/20-037E/