Following , using regions improve the model performance in semantic segmentation tasks.
One naïve idea is, to look at a region (patch) around a pixel and look at the histogram of labels in the patch.
i.e., current loss where p is all the pixels in the image, y is one-hot labels.
new loss: same good old CE, but for each class where n is the number of pixels at the ‘neighborhood.’
This technique called label smoothing and might help our model .
To the extant, we can use weakly supervision, where we get an OD model (trained of some other task) and use the ‘neighborhood’ as the pixels exist in the BBOX of the object.
This might be a problematic method and we need to reshape the bg class with some focal loss.
 Region Mutual Information Loss for Semantic Segmentation: https://arxiv.org/pdf/1910.12037.pdf
 When Does Label Smoothing Help?:
In this project you will: represent the dataset as a graph,
1. where the graph contain cliques for each class and the nodes embedding are a CNN representation of each example.
2. Where the weights of the edges are a score function based on the class similarity (for cifar100 and beyond).
The Stanford Neuromuscular Biomechanics Lab has developed OpenCap, an open-source software for capturing and analyzing human motion data, with the main module, OpenCap Core, serving as a flexible platform that can work with various motion capture systems and data processing tools. OpenCap utilizes two or more IOS-based device cameras to create 3D motion capture, employing OpenPose, an AI-based, markerless 2D pose estimation software. Currently, OpenCap only supports IOS devices. Many action cameras, like GoPro, exhibit a “fisheye” effect due to their wide-angle lens, causing image distortion but providing an extensive field of view. GoPro cameras have different field of view settings to control this effect, and they also offer software for post-production correction. The aim is to adapt OpenCap to GoPro to aid the IQFoil team’s surfers in video analysis.
The adoption of advanced machine learning (ML) methods in physics has led to far-reaching advances in both theoretical predictions and experiments. Nevertheless, there are still physical phenomena, particularly in quantum physics, that have not yet benefited from this progress. One important branch of quantum physics that might benefit significantly from the adoption of ML algorithms is quantum optics. Quantum optics has proven to be an invaluable resource for the realization of many quantum technologies, such as quantum cryptography, sensing, and computing.
If we wish to employ learning-style optimization methods (ML/DL) for problems in quantum physics, it is crucial to have a good physical model of the quantum process in question and integrate it into the algorithm itself, which may be difficult to model when we are discussing problems from quantum mechanics. In a recent paper , we show how to employ machine learning algorithms for inverse design problems in quantum optics. Specifically, we developed an algorithm for generating high-dimensional spatially entangled photon pairs, by tailoring the nonlinear interactions of light. The work has generated a lot of interest [a, b, c] due to its potential to advance many areas in quantum optics. For example, the high dimensionality of these generated states increases the bandwidth of quantum information and can improve the security of quantum key distribution protocols.
In this project, we will refine and improve the algorithm beyond recognition in order to solve new important problems in quantum optics; such as applications in Metamaterials , improving the fidelity in quantum communication, or designing optical interaction that generates a wide realization of maximally entangled high-dimensional states. We will do this by using deep learning tools that replace internal modules based on optimization or numerical methods, and by applying advanced learning techniques such as Neural ODEs , and Implicit Neural Representations .
Disasters like floods and earthquakes threaten human safety, infrastructure, and natural systems. Every year, disasters kill an average of 60,000 people, affect 200 million and cause $150 billion (USD) billion in damage. A timely and accurate understanding of the environment plays a key role in disaster preparedness, assessment, and response. Recently, unmanned aerial vehicles (UAV) with inexpensive sensors have emerged as a practical tool to collect situational imagery from disaster areas that can be hard-to-reach for humans. However, UAVs are equipped with energy-constrained supplies and low-compute devices, which limit the ability to perform automatic on-device analysis. This adds to on-board system latency, resulting in longer response times for disaster relief efforts. Therefore, achieving effective on-device computer vision with low power consumption and low latency remains a significant challenge.
In this project you will explore efficient implementations of semantic segmentation models collected from UAV and deployment of Jetson Nano.
Activision functions are a crucial part of deep neural networks, as a nonlinear transformation of the features.
In this project we will aim to burn an FPGA accelerator and measure the performance of different activations, including some that we developed here in the lab, to help the algorithm developers and see another perspective of choosing the architecture.
BERT , is the most popular transformer model.
Transformers, and Large Language models in particular, are very expensive to run and use.
One simple way to make this models faster it quantization .
In this project we will aim to quantize the Bert model by quantization aware training, with a new method called AMED[*], based on code provided with QBERT [3,4] and by the instructor.
[*] paper and code will be provided after submission to the project.
Instructor: Moshe Kimhi and Dr. Chaim Baskin
Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning (e.g. 236781) and passion for efficient AI.