What are Neural Networks?Neural networks are a series of algorithms that aim to imitate the human brain in order to identify patterns from data. They process information using machine perception by grouping or labeling raw input data.
Consider the complexity of the human brain, which is composed of a network of neurons. It has the remarkable ability to quickly grasp the context of various scenarios, something that computers struggle to do.
Artificial Neural Networks are designed to address this limitation. Initially created in the 1940s, Artificial Neural Networks seek to mimic the functioning of the brain. Sometimes referred to as perceptrons, an Artificial Neural Network is a hardware or software system. It consists of a layered network designed to emulate the operations of brain neurons.
The network includes an input layer for data processed entry and an output layer for presenting information. Connecting the two is a hidden layer, or layers, comprised of units that transform input data into useful information for the output layer.
In addition to emulating human decision-making processes, Artificial Neural Networks enable computers to learn. Their structure allows ANNs to efficiently and effectively identify complex patterns that may be challenging for humans to discern. Furthermore, they enable us to rapidly classify and categorize large volumes of data.
How do Biological Models of Neural Networks Work?
What aspects of human brain structure do neural networks imitate, and how does the training process function?
All mammalian brains are made up of interconnected neurons that transmit electrochemical signals. Neurons have various components: the body, which includes a nucleus and dendrites; axons, which connect to other cells; and axon terminals or synapses that transmit information or stimuli from one neuron to another. Together, they carry out communication and integration functions in the nervous system. The human brain possesses a vast number of processing units (86 billion neurons) that facilitate the performance of highly intricate functions.
How do Artificial Neural Networks Work?
Artificial Neural Networks consist of several layers, each containing artificial neurons known as units, which process, categorize, and organize information. The layers are accompanied by processing nodes, each holding specific knowledge, including programmed rules and learned rules, allowing the network to learn and react to various types of data. Most artificial neural networks are fully connected across these layers, with weighted connections determining the influence between units.
The input layer receives information in various forms, which then progresses through hidden layers for analysis and processing. This processing helps the network learn more about the information until it reaches the output layer, where it works out responses based on the learned information. ANNs are statistical models designed to self-adapt and understand concepts, images, and photographs using learning algorithms.
For processing, developers arrange processors in parallel-operating layers: input layer, hidden layer, and output layer, analogous to the dendrites, cell body, and synaptic outputs in the human brain’s neural network, respectively. The hidden layer uses weighted inputs and a transfer function to generate output.
Various types of Neural Networks
The recurrent neural network, a commonly used type, allows data to flow in multiple directions, enabling complex tasks such as language recognition. Other types include convolutional neural networks, Hopfield networks, and Boltzmann machine networks, each suited for specific tasks based on the entered data and application. More complex tasks may require the use of multiple types of ANN.
Tesla is betting big on autonomy based on neural networks with an impressive showcase.
Today, Tesla hosted an “Autonomy Investor Day” at their headquarters in Palo Alto, CA. At the event, Tesla detailed its plans for advanced driver assistance and eventual car autonomy. The presentation delved into more technical details than previous Tesla disclosures, significantly improving my perception of Tesla’s methods and prospects. This was undoubtedly Tesla’s most significant press event to date.
Unlike most companies working on fully autonomous vehicles, Tesla has taken a distinctive approach. The company plans to rely solely on radar and an array of video cameras around the vehicle to accomplish this.
Most other teams also use these technologies, but supplement them with LIDAR (laser) sensors, which provide the vehicle with exceptional 3-D vision regardless of lighting conditions. During the presentation, Tesla provided a more in-depth explanation of why it has chosen this approach and its criticisms of alternative approaches.
Not only did Tesla express disagreement with other methods, but Elon Musk also derided LIDAR as a “fool’s errand” and asserted that those who depend on it are “doomed.” He also predicted that all other players “will dump LIDAR, mark my words .” Similar sentiments were expressed regarding the use of detailed “HD” maps to understand the road based on previous trips over it.
In essence, Tesla is making a substantial bet that they can address all self-driving challenges using neural networks. They believe that neural network approaches are indispensable for solving the problem, asserting that other methods, including additional sensors like LIDAR, are distractions and unnecessary expenses.
If this bet proves successful, it will be a significant triumph, potentially positioning Tesla as the leader in what is perhaps the most substantial opportunity in modern industry.
There is a lot to dissect from this presentation, and more articles on this topic will follow.
New Chip
Tesla has developed its own custom chip tailored for the specific processing needs of their vehicles, and they are now integrating this chip into all new cars. They are convinced that it provides all the computing power necessary for full self-driving. The chip was designed to dedicate its silicon exclusively to driving-related tasks and to keep power consumption under 100 watts to avoid affecting the vehicle’s range.
The majority of the chip is allocated to conducting dot products for neural network convolutions. Musk contends that this chip surpasses all others globally in terms of neural network capabilities, a claim that may be disputed by other companies developing similar chips. Tesla primarily compared its performance to NVIDIA’s general-purpose GPU chips.
The hardware boasts impressive specifications and is likely adequate for the required computations. While similar chips may become available from other providers, Tesla anticipates that designing their own chip and integrating it into millions of cars will yield long-term cost savings, even factoring in development In addition to the neural network hardware, the chip features a mid-level GPU and 12 64-bit ARM cores for general-purpose computing. The hardware is designed with redundancy to withstand the failure of any component.
Network training
Tesla has focused on enhancing its neural networks with its new network hardware, emphasizing the training of better neural networks to categorize objects encountered on the roads. The company believes its competitive advantage lies in the extensive fleet of cars, currently amounting to around half a million cars, which they utilize for network training.
Andrej Karpathy outlined some of the strategies they employed. Initially, they trained their networks using human-labeled images, and when they encountered something they wanted to improve network training on, they requested their car fleet to upload relevant images, enabling them to amass thousands of images for training data to enhance network performance.
Their approach encompassed various stationary and moving objects and also involved identifying patterns of movement, such as requesting examples of cars cutting in front of Tesla cars. This enabled them to analyze pre-cut-in video footage to train the network to predict future car activities on the road.
They also applied this methodology to path planning, observing human drivers’ path choices in different road scenarios to understand typical human responses. In cases where errors were observed, they prioritized obtaining better data to network enhance training.
Additionally, they achieved significant success in training their networks to estimate distances to objects in the field of view. One method involved leveraging car radars, which provided precise distance measurements to all radar targets. By correlating radar targets with visual targets, they trained the network to estimate distances to visual targets accurately.
Tesla’s extensive fleet of drivers granted them immediate access to new data relevant to their team. It is important to note that any entity with a vast network of dashcam recordings could potentially leverage this approach, although accessing radar data might be a limitation. This type of data is available to multiple parties should they choose to record it. However, Tesla can more effectively manage its fleet due to its regular software updates across all its cars.
This approach has empowered Tesla to establish a robust system for training neural networks for perception and driving. The pivotal question revolves around whether this approach is adequate to achieve the utmost reliability, often referred to as the “final 9s,” necessary to eliminate the car’s steering wheel. Tesla contends that reaching this extremely high level of reliability requires extensive training data, an area in which they have a competitive edge with their large fleet. While it is widely acknowledged that more data is beneficial, there is ongoing debate on whether it is sufficient or if additional techniques are imperative to achieve such an exceptional level of reliability.
Managing software
Tesla has implemented this approach with its recent update for “Navigate on Autopilot,” allowing the vehicle to make lane changes automatically. Initially, this feature required drivers to confirm each lane change. Tesla analyzed drivers’ responses to suggested changes and used the data to improve the system. With automatic lane changes, the system now receives feedback on 100,000 automated changes daily, reporting no accidents related to these maneuvers.
The company also intends to apply this method to enhance its automatic emergency braking (AEB) system to anticipate potential obstacles, including pedestrians, cyclists, and sudden lane intrusions, by the end of this year.
Comparison: Tesla vs. Industry
The main focus of the entire presentation revolved around Tesla’s distinct choice to forego the use of both LIDAR technology and detailed high-definition maps, unlike most other major players in the industry. by other companies.)
The decision by Tesla not to utilize LIDAR has sparked controversy. Though Musk’s viewpoint that LIDAR is a crutch represents a minority stance, the company has presented a compelling argument in support of this position. For a more in-depth analysis of this pivotal issue of cameras versus LIDAR, refer to my detailed article on the matter.
In summary:
1. LIDAR provides consistent visibility in all lighting conditions, while camera views are heavily influenced by factors like day/night variations, weather, and the sun’s position.
2. LIDAR offers true 3D perception, whereas cameras rely on software to interpret the scene and determine the spatial positioning of objects.
3. LIDAR observes the environment at shorter ranges and lower resolutions.
4. Although LIDAR is considerably more expensive, its cost is rapidly decreasing. However, it is not yet commercially available in sufficient quantities and quality levels, except for Waymo. In contrast, cameras are highly affordable.
5. The reliability of computer vision required for camera-based systems to enable self-driving capabilities is not currently at an adequate level, although many are optimistic about imminent breakthroughs.
6. LIDAR alone is insufficient for certain scenarios, such as accurately identifying road debris, traffic signals, and distant objects. tested, extensive computer vision capability is essential.
Tesla Network
Elon Musk presented on the upcoming Tesla network, which I will provide a more detailed account of tomorrow. Users will have the ability to set specific times and regulations governing the use of their vehicles by others.
Initial key points:
Tesla has pledged to eventually establish a ride-hailing service, resembling Uber in appearance, where Tesla owners’ private vehicles will operate in autonomous mode, generating income for the owner. For instance, owners could designate their car as available for the next 5 hours , after which it would join the network and provide rides before returning. They have projected that this service could be available in just 3 years, significantly increasing the value of each Tesla due to its potential revenue-generating capability.
The extent of interest in this option remains uncertain, as well as how many owners will keep their vehicles prepared for immediate deployment to serve others. (Many people store personal items in their cars and may be unwilling to deplete the battery suddenly.) For those who do opt for this, the car will naturally incur expenses and depreciation, estimated at around 37 cents per mile, but Tesla anticipates it could be reduced to 18 cents per mile with their vehicle. Tesla forecasts a network cost of $1 per mile, which is half of Uber’s, but final conclusions have not been reached.
Tesla is highly committed to this concept. In fact, Musk has announced that they will start encouraging customers to purchase the lower-end “Standard Plus” Model 3 instead of the long-range Model 3, as they are constrained by the number of batteries they can produce.
Selling cars with smaller batteries means they can sell more cars, leading to an increased number of vehicles for their future robotaxi service. Musk was questioned about Tesla’s spending on Autonomy and he stated “It’s essentially our entire expense structure,” indicating a significant investment in this plan.
This year, Tesla acquired over $2 million worth of lidar sensors from Luminar. Despite Elon Musk’s disdain for lidar, which he has previously described as a “crutch” and indicated that companies relying on lidar for autonomous capabilities were “doomed,” Tesla appears to be stockpiling these sensors.
Luminar, an Orlando-based lidar manufacturer, revealed in its quarterly earnings report that Tesla was its “largest LiDAR customer in Q1,” accounting for over 10 percent of the company’s revenue for the quarter, which amounts to approximately $2.1 million worth of lidar based on Luminar’s $21 million quarterly revenue. This substantial purchase from Tesla helped offset a decrease in revenue driven by a reduced volume of sensors supplied to non-automotive companies. However, it was not enough to prevent Luminar from announcing layoffs affecting around 20% of its workforce, and Tesla also initiated employee layoffs.
This marks a significant turnaround for Tesla, as the company has significantly reduced the number of sensors it uses to power advanced driver-assist features like Autopilot and Full Self-Driving over the years. These are features that Musk has consistently positioned as a precursor to a fully autonomous vehicle fleet. It is expected that Tesla will unveil a robotaxi prototype later this year, a project on which Musk is staking the future of the company.
Musk’s aversion to lidar was evident during Tesla’s recent quarterly earnings call, during which he emphasized the reliance on camera-based vision systems to power the vehicles’ driver-assist features and boasted about the potential for achieving self-driving with a relatively low-cost inference computer and standard cameras, without the need for lidars, radars, or ultrasonic sensors.
The purpose of Tesla’s acquisition of $2.1 million worth of Luminar lidar sensors remains unknown. Luminar spokesperson Milin Mehta declined to comment, and Tesla has not formally responded to any reporters’ inquiries since 2019.
Nevertheless, it should not be entirely surprising that Tesla is showing interest in lidar technology. In 2021, a Tesla Model Y was spotted in Florida with rooftop lidar sensors manufactured by Luminar. Additionally, Bloomberg reported that Tesla had partnered with Luminar to utilize lidar for “testing and developing,” although the specifics of this collaboration remain undisclosed.
When questioned in 2021 about the Tesla deal, Luminar founder and CEO Austin Russell declined to comment, citing “customer confidentiality.” He mentioned that Luminar sells its older Hydra lidar units to certain customers for “testing, development, data collection, [and] benchmarking.”
Even if Tesla is using Luminar’s lidar to validate its Full Self-Driving feature for an upcoming robotaxi launch, that’s still a substantial amount of lidar. According to Luminar, individual lidar sensors cost around $1,000, including software. Could it be that Tesla purchased 2,100 lidars for its vehicles? Possibly! The company is quietly operating an autonomous testing fleet in multiple cities, including San Francisco and Las Vegas. Will it retrofit those company-owned vehicles with Luminar’s lidar? If it does, people will take notice, just like they did with the one Model Y in Florida several years ago. We will soon find out whether those vehicles are ready to hit the road.
In response to a Musk-fan account mocking this article on X, Musk stated that Tesla didn’t require the lidar for validation purposes, without clarifying the purpose of the sensors.
What does appear evident is that Tesla is shifting its stance on lidar, even if Musk publicly remains opposed to it. Eventually, the CEO himself may be compelled to set aside his pride and acknowledge that lasers are indeed valuable.
NHTSA reports that at least 20 vehicle crashes occurred after Tesla recalled 2 million vehicles with Autopilot. The government is seeking to understand the reasons behind this.
Following Tesla’s voluntary recall of 2 million vehicles with Autopilot, there have been at least 20 crashes involving Tesla vehicles with Autopilot engaged. The National Highway Traffic Safety Administration (NHTSA) disclosed this information in a recent filing.
Tesla issued a recall for over 2 million vehicles with Autopilot in response to NHTSA’s investigation into numerous crashes involving the company’s driver-assist feature, including several fatal ones. The recall aimed to address concerns related to driver inattention and Tesla’s warning systems, which NHTSA stated have contributed to hundreds of crashes and dozens of fatalities. However, last month, the agency initiated a new investigation into Tesla’s fix and is now requesting additional information from the company.
In its request for information, NHTSA mentioned that a preliminary analysis revealed at least 20 crashes in Tesla vehicles equipped with the updated version of Autopilot. Of these crashes, nine involved Teslas colliding with other vehicles or pedestrians in their path — termed “frontal plane” crashes by the agency. These crashes suggest that Tesla’s camera-based vision system may be insufficient in detecting certain objects in front of the vehicle when Autopilot is engaged.
NHTSA is asking Tesla to provide data that will enable its investigators to compare vehicle performance in these types of crashes before and after the recall, including the number of “Hands-on-Wheel” warnings issued to drivers. Last month, NHTSA criticized Tesla’s ” weak driver engagement system with Autopilot’s permissive operating capabilities.”
Other details requested by NHTSA include explanations for Tesla’s one-week suspension policy for misuse of Autopilot, driver monitor warnings, driver-facing alerts, and the single pull versus double pull of the driver stalk to activate Autopilot. NHTSA is also seeking information about ” Tesla’s use of human factor science in its design,” including the number of employees dedicated to these designs.
NHTSA is requesting data from Tesla regarding the collection of telemetry data following crashes that happen when the vehicle is in Autopilot or Full Self-Driving mode. Additionally, it is seeking more information about how Tesla utilizes the in-cabin camera to monitor driver attention. The agency warns that failure to comply with its information request could result in Tesla facing fines of up to $135 million. Tesla has time until July 1st, 2024, to provide the requested information.
Elon Musk, the CEO of Tesla, has previously expressed his opinion that lidar sensors are a crutch for autonomous vehicles. Nevertheless, Tesla has become the top customer of the lidar manufacturer Luminar after purchasing a significant number of lidar sensors from the company.
Luminar recently revealed in its first-quarter earnings report that Tesla contributed to over 10% of its revenue in the first quarter of 2024, totaling a little more than $2 million. Despite a 5% decline in revenue from the previous quarter, mainly attributed to reduced sensor sales to non-automotive clients, Luminar’s revenue was bolstered by increased sensor sales to Tesla, its largest lidar customer in Q1. Luminar also noted a 45% year-over-year revenue gain.
During the first quarter, Luminar reported a net loss of $125.7 million, an improvement compared to the $146.7 million loss reported during the same period the previous year. The company attributed its net loss to accelerated depreciation for equipment expected to be abandoned following certain outsourcing actions initiated in fall 2023.
In recent news, Luminar announced plans to reduce its workforce by 20% and outsource a significant portion of its lidar sensor production as part of a restructuring effort to scale the business.
Tesla has been observed using lidar and other sensors on its test vehicles, and there have been reports of a partnership with Luminar dating back to 2021. However, details of this collaboration have never been disclosed. Luminar included Tesla in its earnings report in line with historical SEC guidance, revealing the information just prior to Tesla’s anticipated reveal of a robotaxi on August 8.
Elon Musk has consistently argued against the use of lidar for autonomous vehicle navigation, stating that it is an unnecessary and expensive sensor. Musk previously asserted at Tesla’s “Autonomy Day” event in 2019 that relying on lidar is futile and akin to having multiple unnecessary appendices .
Musk also mentioned at the same event in 2019 that Tesla would launch a fleet of robotaxis within a year, a promise that did not materialize. Instead, Tesla’s involvement in purchasing lidar sensors continues.
The term “lidar” stands for light detection and ranging and was initially developed alongside the invention of lasers in the 1960s. While it was intended to play a significant role in the advancement of autonomous vehicles, negative remarks from the leader of a prominent autonomous vehicle company were not favorable for the Lidar technology sector.
Chinese car manufacturers are at the forefront of the shift towards Lidar technology in the automotive industry.
In 2023, more new cars were equipped with Lidar compared to the previous four years, with Chinese automakers leading this trend. Analysts at the Yole Group predict that around 128 car models with Lidar will be launched by Chinese manufacturers this year, surpassing the expected releases in Europe and the US.
The cost of Lidar technology in Chinese cars has substantially decreased, with an average price of USD 450-500, compared to the global average of USD 700-1000. The global market for Lidar in passenger cars, light commercial vehicles, and robotaxis was estimated to be USD538 million in 2023, marking a 79% increase from the previous year.
Although more passenger cars are currently integrating Lidar compared to robotaxis, this gap is expected to narrow as the market continues to expand. Japanese and South Korean car manufacturers are also likely to introduce car platforms with Lidar in 2024 or shortly thereafter. The decreasing cost of Lidar technology has facilitated its adoption in lower-priced car segments.
This trend highlights how certain technologies may take time to mature but can experience rapid growth once their moment arrives. For example, QR code technology only gained prominence in Australia after the COVID-19 lockdowns, and Bluetooth technology, developed by Hedy Lamarr in 1941, became widely utilized in recent decades.
Despite Elon Musk’s previous skepticism, he has now begun integrating Lidar into vehicles, although without a full endorsement. Lidar, which stands for “Light Detection and Ranging”, utilizes laser projections to create detailed real-time maps of the surrounding environment. Besides aiding autonomous vehicles, Lidar is used for creating precise 3D scans of various landscapes and structures.
Furthermore, it played a role in the production of Radiohead’s House of Cards music video. When mounted on a vehicle, Lidar can generate accurate 3D maps of the surroundings up to 60 meters in all directions, enhancing the vehicle’s ability to detect obstacles and avoid collisions Despite its cost, Lidar provides visibility in scenarios where other sensors may fall short.
“Lidar is a hybrid technology, situated between cameras and radar, that can detect distance and objects while discerning the shape of those objects,” said Richard Wallace, who leads the Transportation Systems Analysis group in the Center for Automotive Research.
Cameras and radar, both employed in the Tesla Model S, have their limitations, Wallace noted. “Cameras, like our eyes, rely on optics. In low light or during a blizzard, cameras struggle.”
On the other hand, radar excels at detecting objects and their distance but cannot provide information on the shape or size of the object. The radar in the Model S likely detected the truck it collided with, but it is programmed to ignore objects that resemble overhead road signs to avoid “false braking events.”
“They have to do that, otherwise imagine going down a highway and every time you come to an overpass it hits the brakes,” Wallace explained. “Clearly the algorithm needs some refinement.”
While appreciative that the Model S is not designed to be fully autonomous, Wallace suggested that Tesla may need to reconsider its stance on Lidar to achieve its self-driving ambitions.
“I know Elon Musk has said Lidar isn’t necessary. He’s obviously a smart guy, but ultimately, I believe it will be proven that Lidar is needed,” he said. “It adds a level of resiliency and redundancy that makes the integration easier to solve.”
The integration Wallace refers to involves the algorithms and intelligence that coordinate the function of the various sensors. “All sensors have their own limitations. How can you create the brain that integrates them and makes the correct decisions?”
Wallace believes that lidar and vehicle-to-vehicle communication, where each car communicates its location to others nearby, will both be crucial in building safer self-driving fleets.
Google uses Lidar units that cost up to $70,000 in its self-driving cars, although there are now units available for as little as $250. This could potentially make Lidar more accessible for the mass market.
However, simply having Lidar does not guarantee the safety of a driverless car. Google’s fleet has experienced its fair share of accidents and technical issues, although there have been no reported fatalities to date.
Tesla declined to comment but referred the Guardian to Musk’s previous comments about Lidar not being necessary for driverless navigation. The company also pointed to a list of factors in the Model S user manual that can impede the performance of autopilot, including poor visibility, bright light , damage or obstructions caused by mud, ice, snow, and extreme temperatures.
The list of limitations is accompanied by a warning stating: “Never depend on these components to keep you safe. It is the driver’s responsibility to stay alert, drive safely, and be in control of the vehicle at all times.”
The company also directed readers to a blogpost titled Your Autopilot Has Arrived, which asserts: “The driver is still responsible for, and ultimately in control of, the car. What’s more, you always have intuitive access to the information your car is using to inform its actions.”
Understanding the construction of a LiDAR system
A LiDAR system requires specific equipment to measure a million distances from sensors to surface points. It operates at a high speed, capable of calculating distances based on the speed of light, which is 300,000 kilometers per second. In various applications, including automotive vehicles, aircraft, and UAVs, LiDAR systems consist of three main components:
Laser Scanner
LiDAR systems emit laser light from different mobile platforms like automobiles, airplanes, and drones, and receive the light back to measure distances and angles. The scanning speed significantly impacts the number of points and echoes recorded by a LiDAR system, while the choice of optic and scanner profoundly influences its resolution and operating range.
Navigation and positioning systems
It is essential to determine the absolute position and orientation of a LiDAR sensor when mounted on aircraft, a vehicle, or an unmanned aerial system (UAS) to ensure the usefulness of the captured data. Global Navigation Satellite Systems (GNSS) provide accurate geographical information about the sensor’s position (latitude, longitude, height), while an Inertial Measurements Unit (IMU) precisely defines the sensor’s orientation (pitch, roll, yaw) at this location. The data recorded by these devices are then used to create static points comprising the basis of the 3D mapping point cloud.
Computing technology
Computation is necessary for a LiDAR system to define the precise position of echoes and make the most of the captured data. It is used for on-flight data visualization, data post-processing, and to enhance precision and accuracy in the 3D mapping point cloud.
Matching project needs with LiDAR specifications
Laser Scanner: Evaluate the accuracy, precision, point density, range, and swath that best suits your project requirements.
GNSS: Assess the compatibility of the GNSS reference station (terrestrial) and GNSS receiver (moving) with the GNSS used (GPS, GLONASS, BEiDOU, or Galileo) and determine if a ground station is needed.
Batteries: Determine if the LiDAR system uses internal or external batteries and the required autonomy to cover the intended mapping area.
Mounting: Consider if the LiDAR system can be easily mounted on the aerial/airborne platform (drone, aircraft) or automotive platform (vehicle) you intend to use.
Datafile: Look into the format of the generated data file, for example, YellowScan LiDAR models associated with CloudStation software can export point clouds as .LAZ or .LAS files, as well as digital terrain or elevation models.
Data Post-processing: Assess the ease of using the data and delivering the best 3D mapping point cloud to your end customer. Consider classification, colorization using additional high-resolution cameras, DTM generation, and what to do with the post-processed data.
Uncovering applications of LiDAR on UAVs
Energies & Utilities: conducting powerline surveys to identify sagging issues or plan trimming operations
Mining: undertaking surface/volume calculations to enhance mine operations (stockpile, excavation) or decide on mine extension
Construction & engineering: creating maps for leveling, planning, and infrastructure optimization (roads, railways, bridges, pipelines, golf courses) or renovating post natural disasters, conducting beach erosion surveys to develop emergency plans
Archaeology: mapping through forest canopies to accelerate discoveries of objects
Forestry: mapping forests to optimize activities or assist in tree counting
Environmental research: measuring growth speed and disease spreading
Exploring the use of UAV for LiDAR mapping
- Learn more about DJI UAVs for LiDAR mapping such as DJI M600 or DJI M300.
- Selecting the appropriate UAV for your next LiDAR surveys is a challenging task. Read further about how to select your UAV to commence your LiDAR operations.
- Discover the crucial aspects of a good UAV LiDAR integration or some instances of integrating our LiDAR models on drone or airborne platforms.
Is it possible for LiDAR to penetrate through trees?
LiDAR systems with multiple returns and higher pulse rates can aid in reducing the impact of vegetation interference. Additionally, specialized processing methods can be utilized to filter out foliage and generate more precise ground elevation models. While LiDAR can offer valuable insights even in vegetated areas, its effectiveness relies on the specific conditions and technology used.
Can LiDAR be employed for scanning in low light?
Indeed, LiDAR can be utilized for scanning in low light since it does not rely on visible light like conventional cameras. LiDAR systems emit their own laser pulses, which are then reflected off objects and returned to the sensor. The system measures the time it takes for the pulses to return, enabling the creation of a detailed 3D map of the environment, irrespective of ambient light conditions.
This functionality makes LiDAR particularly useful for tasks such as autonomous driving vehicles, surveillance, and navigation under low-light or nighttime conditions. Moreover, LiDAR is increasingly utilized in the consumer market, as seen in Apple’s iPhone. The integration of LiDAR technology into the iPhone’s camera results in faster, more accurate autofocusing, particularly in low-light conditions, contributing to the delivery of sharp, focused images even in challenging lighting situations.
How does LiDAR identify objects?
LiDAR identifies objects through the emission of rapid laser pulses and the use of sensors to measure the time it takes for those pulses to bounce back after hitting surfaces. The system calculates the distance based on the time delay, creating a point cloud that represents the shape and position of the object in 3D space. This enables accurate object detection and mapping in various applications such as autonomous driving vehicles, environmental monitoring, and others. The point cloud can also be utilized to generate a digital elevation model (DEM) or a digital terrain model (DTM).
Can LiDAR penetrate through the ground?
LiDAR is capable of penetrating the ground to some extent, depending on the material and conditions. The ability of LiDAR to penetrate the ground is constrained by factors like the type and thickness of the material. For instance, LiDAR can penetrate vegetation or even water, employing bathymetric lasers to measure underwater surface depth. However, dense soil or rock cannot be penetrated by LiDAR. Ground-penetrating radar (GPR) is a distinct technology designed specifically to penetrate the ground and provide information about subsurface structures, functioning on different principles compared to LiDAR scanning.
At what range is LIDAR accurate?
The accuracy of LiDAR can vary based on several factors, including the type of LiDAR system, the technology utilized, the quality of the equipment, and the specific application. Generally, LiDAR is renowned for its high accuracy in measuring distances, often achieving sub-centimeter to centimeter-level accuracy under favorable conditions.
For airborne LiDAR systems, commonly employed for mapping large areas, the accuracy can be maintained even at longer distances. High-end airborne LiDAR systems can attain accuracies of a few centimeters at distances ranging from tens to hundreds of meters.
It’s essential to note that accuracy can be influenced by factors such as atmospheric conditions, the reflectivity of the surfaces being measured, and the quality of the LiDAR equipment. Calibration, data processing, and correction techniques in software also play a critical role in achieving accurate results.
Self-Driving Cars
What embodies the “future” more than a self-driving car? Over the past 30 years, we’ve envisioned cyberpunk dystopian worlds where androids dreaming of electric sheep evade captors by boarding driverless vehicles. Perhaps these vehicles could fly, but you understand the point.
Autonomous vehicles are no longer just a dream. While most of them are still in prototype stage, they are unquestionably a reality today. Numerous companies
Artificial Neural Networks in Financial Services
In the realm of AI banking and finance, Artificial Neural Networks are well-suited for making predictions. This capability is largely due to their capacity to swiftly and accurately analyze vast amounts of data. Artificial Neural Networks can process and interpret both structured and unstructured data . Once this information is processed, Artificial Neural Networks can make precise forecasts. The accuracy of the predictions improves as more information is provided to the system.
Enhancing Operational Efficiency of Banks
The predictive capabilities of Artificial Neural Networks are not limited to the stock market and exchange rate scenarios. These capabilities also have applications in other areas of the financial sector. Mortgage assessments, overdraft calculations, and bank loan evaluations are all based on the analysis of an individual account holder’s statistical information. Previously, the software used for this analysis was driven by statistics.
Banks and financial providers are increasingly transitioning to software powered by Artificial Neural Networks. This shift enables a more comprehensive analysis of the applicant and their behavior.
As a result, the information presented to the bank or financial provider is more accurate and valuable. This, in turn, allows for better-informed decisions that are more suitable for both the institution and the applicant. According to Forbes, many mortgage lenders anticipate a surge in the adoption of systems powered by Artificial Neural Networks in the coming years.
Tesla has been making promises regarding its Full Self-Driving (FSD) capability for some time, even selling a beta version to customers willing to purchase the software. FSD is marketed as a more advanced option compared to its Autopilot and Enhanced Autopilot driver assistance features.
Often characterized as the more sophisticated but still experimental component of Tesla’s driver assistance lineup, FSD includes what the company refers to as Autosteer on City Streets along with Traffic and Stop Sign Control.
The most recent update, version 12.1.2, stands out from earlier iterations due to one significant change.
“FSD Beta v12 enhances the city-streets driving technology by implementing a single, comprehensive neural network trained using millions of video clips, thus replacing over 300k lines of dedicated C++ code,” Tesla noted in its release documentation.
Neural networks, commonly known as artificial neural networks (ANNs), are generally described as a form of machine learning technology that improves its efficiency and accuracy through training data over time. In Tesla’s application, these neural networks have been educated using actual video footage to make decisions instead of relying on extensive lines of code.
The introduction of neural networks in this FSD beta update marks a new direction for the automaker, which has shifted to a vision-exclusive method for its software and sensor configuration in recent years, moving away from the combination of vision, radar, and lidar used by competitors working on autonomous technologies.
This transition to a neural network-centric approach in FSD beta reinforces Tesla’s commitment to a vision-only sensor setup, which helps clarify the decision to eliminate other sensors a couple of years back.
The efficacy of the latest beta version in delivering enhancements remains uncertain, but numerous overarching questions still linger regarding FSD.
For example, it hasn’t become any clearer over time to pinpoint exactly what Tesla envisions FSD will ultimately provide.
“Full autonomy will depend on achieving reliability that far surpasses human drivers, as evidenced by billions of miles of driving experience, along with obtaining regulatory approval, which may vary in timing by region,” Tesla states concerning its three systems, while deliberately avoiding the SAE level classification.
Previously, Tesla has informed California regulators that FSD’s capabilities do not exceed SAE Level 2.
If this still holds true, it makes sense from a regulatory standpoint, as SAE Level 3, often defined as systems allowing the driver to disengage from active monitoring, are currently allowed only in a select few states. This has already resulted in considerable challenges for European and Japanese automakers who have implemented such systems in other markets but cannot do so across all states in the U.S.
These SAE Level 3 systems permit drivers to look away from the road for extended periods, enabling them to read, watch videos, or respond to emails—capabilities that FSD does not currently permit.
“Always keep in mind that Full Self-Driving (Beta) does not make Model Y autonomous and necessitates that the driver remains fully attentive, ready to act instantly at any moment,” Tesla clarifies on its site.
If FSD were to suddenly acquire the capability to function for hours without the need for driver intervention or even attention to external conditions, Tesla could face substantial regulatory challenges in the majority of U.S. states and would have to acknowledge it as a Level 3 system.
A more pressing concern is that Tesla has spent five years refining what still appears to be a Level 2 system without officially labeling it as such, while other manufacturers, including Mercedes-Benz, have already begun deploying SAE Level 3 systems in select U.S. states as well as abroad.
Tesla has also not disclosed any developments regarding SAE Level 4 robotaxi technology, which it once aimed to achieve, but which has already seen operational rollouts in various U.S. cities by other companies, alongside some setbacks and controversies over the past year.
It’s important to note that all these Level 3 and Level 4 systems utilize more than just vision, incorporating a variety of radar and lidar sensors in addition to cameras.
The future evolution of FSD into a Level 3 system remains uncertain in the coming years, especially as regulators in individual states continue to be cautious about such systems from other manufacturers.
It’s time to explore again how Tesla plans to execute FSD. Once more, a thank you to SETI Park on X for their outstanding reporting on Tesla’s patents.
This time, the focus is on Tesla developing a “universal translator” for its AI, which enables its FSD and other neural networks to seamlessly adjust to various hardware systems.
This translation layer will let a complex neural network—such as FSD—function on virtually any platform that fulfills its basic requirements. This will significantly shorten training times, accommodate platform-specific limitations, and enhance both decision-making and learning speed.
Let’s examine the main points of the patents and simplify them as much as possible. This latest patent is likely how Tesla plans to apply FSD in non-Tesla vehicles, Optimus, and other devices.
Decision-Making
Consider a neural network as a mechanism for making decisions. However, constructing one also involves making a series of choices regarding its design and data processing techniques. Think of it like selecting the right ingredients and culinary methods for a complicated recipe. These selections, known as “decision points,” are vital to how effectively the neural network operates on a particular hardware platform.
To automate these choices, Tesla has created a system akin to a “run-while-training” neural network. This clever system evaluates the hardware’s capabilities and modifies the neural network in real-time, guaranteeing peak performance regardless of the platform.
Constraints
Every hardware platform has its own limitations—such as processing capabilities, memory size, and supported instructions. These limitations serve as “constraints” that determine how the neural network can be set up. Picture it like attempting to bake a cake in a small kitchen with a limited oven and counter space. You must adjust your recipe and methods to suit the constraints of your equipment or environment.
Tesla’s system automatically detects these constraints, enabling the neural network to function within the hardware’s limits. Consequently, FSD could be transferred between vehicles and quickly adapt to a new context.
Now, let’s outline some of the essential decision points and constraints involved:
Data Layout: Neural networks handle extensive amounts of data. The way this data is organized in memory (the “data layout”) greatly influences performance. Different hardware setups may favor distinct layouts. For instance, some may operate more efficiently with data arranged in the NCHW format (batch, channels, height, width), while others may prefer NHWC (batch, height, width, channels). Tesla’s system autonomously chooses the best layout depending on the target hardware.
Algorithm Selection: Numerous algorithms can be employed for functions within a neural network, including convolution, which is vital for image processing. Some algorithms, like the Winograd convolution, offer faster processing but may need specific hardware support. Others, such as Fast Fourier Transform (FFT) convolution, are more flexible but could be slower. Tesla’s system smartly selects the optimal algorithm according to the capabilities of the hardware.
Hardware Acceleration: Contemporary hardware often comes with specialized processors intended to boost the speed of neural network tasks. These include Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs). Tesla’s system detects and leverages these accelerators, maximizing performance on the specific platform.
Satisfiability
To discover the ideal configuration for a specific platform, Tesla utilizes a “satisfiability solver.” This powerful tool, particularly a Satisfiability Modulo Theories (SMT) solver, functions like an advanced puzzle-solving mechanism. It translates the neural network’s requirements and the hardware’s limitations into logical formulas and searches for a solution that meets all constraints. Imagine it as assembling puzzle pieces once the borders (constraints) have been established.
Here’s the process, step-by-step:
Define the Problem: The system converts the needs of the neural network and the constraints of the hardware into a series of logical statements. For instance, “the data layout needs to be NHWC” or “the convolution algorithm must be compatible with the GPU.”
Search for Solutions: The SMT solver navigates through the extensive range of potential configurations, employing logical reasoning to dismiss invalid options. It systematically experiments with various combinations of settings, such as adjusting data layouts, choosing algorithms, and enabling hardware acceleration.
Find Valid Configurations: The solver determines configurations that comply with all constraints. These represent possible solutions to the “puzzle” of efficiently running the neural network on the selected hardware.
Optimization
Identifying a working configuration is just one part of the equation; pinpointing the optimal configuration is the true challenge. This involves optimizing various performance metrics, such as:
Inference Speed: The rate at which the network processes data and renders decisions. This aspect is crucial for real-time functionalities like FSD.
Power Consumption: This refers to the energy utilized by the network. It is crucial to optimize power consumption to extend battery life in both electric vehicles and robots.
Memory Usage: This indicates the amount of memory needed to store the network along with its data. Reducing memory usage is particularly vital for devices with limited resources.
Accuracy: It is critical to ensure that the network retains or enhances its accuracy on the new platform for the sake of safety and reliability.
Tesla’s system assesses potential configurations using these metrics, choosing the one that provides the best overall performance.
Translation Layer vs Satisfiability Solver: It’s essential to differentiate between the “translation layer” and the satisfiability solver. The translation layer encompasses the entire adaptation process, managing components that evaluate the hardware, set the constraints, and call upon the SMT solver. The solver is a specific tool employed by the translation layer to discover valid configurations. You can think of the translation layer as the conductor of an orchestra, whereas the SMT solver is one of the instruments playing a key role in the harmonious adaptation of AI.
Simple Terms: Picture having a complicated recipe (the neural network) and wanting to prepare it in various kitchens (hardware platforms). Some kitchens have a gas stove, while others use electricity; some feature a spacious oven, and others only have a small one. Tesla’s system serves as a master chef, adjusting the recipe and techniques to best suit each kitchen, ensuring a delectable meal (efficient AI) regardless of the cooking environment.
What Does This Mean? To summarize and contextualize this for Tesla—there’s a lot to it. Essentially, Tesla is developing a translation layer capable of adapting FSD for any platform that meets the minimum requirements.
This implies that Tesla can quickly enhance the rollout of FSD across new platforms while identifying the optimal configurations to maximize both decision-making speed and energy efficiency across those platforms.
Overall, Tesla is gearing up to license FSD, indicating an exciting future. This isn’t limited to vehicles; don’t forget about Tesla’s humanoid robot, Optimus, which also operates on FSD. FSD itself may represent a highly adaptable vision-based AI.
What Tesla is Changing to Improve Sentry Mode Efficiency: Recently, Tesla implemented power efficiency upgrades for the Sentry Mode feature of the Cybertruck with software update 2024.38.4. These upgrades significantly enhance the vehicle’s power consumption while Sentry Mode is active.
We now have uncovered more details on how Tesla accomplished such substantial reductions in power consumption, which is estimated to be 40%.
Tesla implemented architectural changes regarding how it processes and analyzes video—optimizing the allocation of tasks among different components. Although the Cybertruck is the first to enjoy these advancements, Tesla intends to roll out these upgrades to other vehicles in the future.
Sentry Mode Power Consumption: Tesla vehicles are equipped with two primary computers: the MCU (Media Control Unit), which drives the vehicle’s infotainment system, and the FSD computer, responsible for Autopilot and FSD functionalities. Both computers remain active and powered whenever the vehicle is awake, drawing around 250-300 watts.
Generally, this power is only utilized when the vehicle is awake or in motion. This is not a major issue as the car automatically enters sleep mode and deactivates its computers after approximately 15 minutes of inactivity. However, the larger concern is that these computers must stay powered on when Sentry Mode is engaged, resulting in a continuous 250-watt draw during this time.
Interconnected System: Currently, the vehicle’s cameras are linked to the FSD computer, which in turn connects to the MCU, followed by the USB ports. Due to this interconnected structure, everything must remain powered. Footage needs to be streamed from the FSD computer to the MCU, where tasks like motion detection take place. The data then has to be compressed before it can finally be recorded on the USB drive. This lengthy process necessitates that multiple computers remain powered to record and save live video.
Architectural Changes: Tesla is implementing architectural modifications to mitigate the high power consumption of Sentry Mode by redistributing tasks among the vehicle’s computers. By reallocating motion detection and possibly compression tasks to the FSD computer, Tesla can now allow the MCU to remain in sleep mode. The MCU is still necessary to transfer the video to the USB drive, but Tesla can wake it up only when it is required.
For example, while the FSD computer will still manage the connection to the vehicle’s cameras, it will also be responsible for detecting motion. When a Sentry event is triggered, it can activate the MCU to save the data to the USB drive and then return it to sleep mode.
This strategy ensures that the MCU does not stay continuously powered for video analysis and compression, activating only when it is needed to manage data.
Processor Isolation & Task Allocation
Tesla’s existing architecture keeps the Autopilot Unit (APU) distinct from the MCU. This separation is motivated by several factors, with safety being the primary concern. The MCU can be rebooted independently during a drive without affecting the APU and crucial safety features.
Furthermore, isolating the APU from the MCU allows tasks that are better suited for each component—such as processing and image transcoding—to be assigned to the appropriate processing unit. This ensures that both the APU and MCU operate at their peak power and performance levels, promoting more efficient energy consumption.
Kernel-Level Power Management
Tesla is focusing on more than just full self-driving (FSD) enhancements or new vehicle visualization updates; they are also optimizing the core kernel of the operating system. Though not extensively employed, Tesla minimizes the clock speed of both the MCU and APU, which leads to lower power consumption and reduced heat output.
Moreover, other kernel enhancements and programming techniques, similar to those Tesla applies to boost the efficiency of its FSD models, contribute to the overall improved efficiency of the vehicles.
Additional Benefits
Given that Tesla vehicles come equipped with a Dashcam that handles video processing, it’s likely that these extra power savings will be observed when the vehicle is operational. This could also influence other functionalities, such as Tesla’s Summon Standby feature, which keeps the vehicle awake and processing video, allowing users near-instant access to the Summon feature of the vehicle.
Roll Out to Other Vehicles
Although the Cybertruck was the first to benefit from these power enhancements in Sentry Mode, it has been indicated that these improvements will be extended to other vehicles as well. Tesla is initially rolling out these changes with the Cybertruck, taking advantage of its smaller user base for preliminary testing before broadening the distribution to other models.
USB Port Power Management
To further enhance energy conservation and reduce waste, Tesla now shuts down USB ports even when Sentry Mode is activated. This adjustment has affected numerous users who depend on 12v sockets or USB ports for powering accessories like small vehicle refrigerators.
It remains unclear if these modifications to Sentry Mode directly influence this change or if the power to the 12v outlets was turned off solely due to safety considerations.
