ONGOING PROJECTS

Projects: Ongoing

Computer Vision

Improve Semantic segmentation via Spatial Label Smoothing

Following [1], using regions improve the model performance in semantic segmentation tasks.
One naïve idea is, to look at a region (patch) around a pixel and look at the histogram of labels in the patch.
i.e., current loss $L=\sum_{p}{CE(y,\hat{y})}$ where p is all the pixels in the image, y is one-hot labels.
new loss: same good old CE, but for each class $y_i^{new}=\sum_{n}\frac{y_i}{n}$ where n is the number of pixels at the ‘neighborhood.’
This technique called label smoothing and might help our model [2].
To the extant, we can use weakly supervision, where we get an OD model (trained of some other task) and use the ‘neighborhood’ as the pixels exist in the BBOX of the object.
This might be a problematic method and we need to reshape the bg class with some focal loss.
[1] Region Mutual Information Loss for Semantic Segmentation: https://arxiv.org/pdf/1910.12037.pdf
[2] When Does Label Smoothing Help?:
https://proceedings.neurips.cc/paper/2019/file/f1748d6b0fd9d439f71450117eba2725-Paper.pdf

Supervisor(s): Dr. Chaim Baskin and Moshe Kimhi

Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning or computer vision is a mush.

Link prediction on Graphs for Semi Supervision

In this project you will: represent the dataset as a graph,
1. where the graph contain cliques for each class and the nodes embedding are a CNN representation of each example.
2. Where the weights of the edges are a score function based on the class similarity (for cifar100 and beyond).

Supervisor(s): Dr. Chaim Baskin and Moshe Kimhi

Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning (e.g. 236781). Advantage: knowledge in graph neural networks.

Adapting 3D Markerless Pose Estimation (OpenCap) to Wider Use

The Stanford Neuromuscular Biomechanics Lab has developed OpenCap, an open-source software for capturing and analyzing human motion data, with the main module, OpenCap Core, serving as a flexible platform that can work with various motion capture systems and data processing tools. OpenCap utilizes two or more IOS-based device cameras to create 3D motion capture, employing OpenPose, an AI-based, markerless 2D pose estimation software. Currently, OpenCap only supports IOS devices. Many action cameras, like GoPro, exhibit a “fisheye” effect due to their wide-angle lens, causing image distortion but providing an extensive field of view. GoPro cameras have different field of view settings to control this effect, and they also offer software for post-production correction. The aim is to adapt OpenCap to GoPro to aid the IQFoil team’s surfers in video analysis.

Supervisor(s): Barak Gahtan, Raz Margi, Asst. Prof. Arielle Fischer

details

Deep Learning Algorithms & Hardware

AI in Quantum Optics

The adoption of advanced machine learning (ML) methods in physics has led to far-reaching advances in both theoretical predictions and experiments. Nevertheless, there are still physical phenomena, particularly in quantum physics, that have not yet benefited from this progress. One important branch of quantum physics that might benefit significantly from the adoption of ML algorithms is quantum optics. Quantum optics has proven to be an invaluable resource for the realization of many quantum technologies, such as quantum cryptography, sensing, and computing.

If we wish to employ learning-style optimization methods (ML/DL) for problems in quantum physics, it is crucial to have a good physical model of the quantum process in question and integrate it into the algorithm itself, which may be difficult to model when we are discussing problems from quantum mechanics. In a recent paper [1], we show how to employ machine learning algorithms for inverse design problems in quantum optics. Specifically, we developed an algorithm for generating high-dimensional spatially entangled photon pairs, by tailoring the nonlinear interactions of light. The work has generated a lot of interest [a, b, c] due to its potential to advance many areas in quantum optics. For example, the high dimensionality of these generated states increases the bandwidth of quantum information and can improve the security of quantum key distribution protocols.

In this project, we will refine and improve the algorithm beyond recognition in order to solve new important problems in quantum optics; such as applications in Metamaterials [2], improving the fidelity in quantum communication, or designing optical interaction that generates a wide realization of maximally entangled high-dimensional states. We will do this by using deep learning tools that replace internal modules based on optimization or numerical methods, and by applying advanced learning techniques such as Neural ODEs [3], and Implicit Neural Representations [4].

[1] https://doi.org/10.1364/OPTICA.451115

[2] https://www.science.org/doi/10.1126/science.abq8684

[3] http://papers.neurips.cc/paper/7892-neural-ordinary-differential-equations.pdf

[4] https://www.vincentsitzmann.com/siren/

[a] https://www.ynet.co.il/environment-science/article/skxiy2r0q

[b] https://www.technion.ac.il/en/2022/10/entangled-photons-computational-learning/

[c] https://quantum.technion.ac.il/Quantum-Entanglement-In-Crystal

Supervisor(s): Prof. Alex Bronstein, Eyal Rozenberg

Requirements: We are looking for someone with high mathematical skills and practical experience in writing ML/DL algorithms (and/or advanced optimization methods). Advantages: interest/experience in the field of physics, Elementary course in Deep Learning (e.g. 236781), experience with PyTorch/Jax.

2023 Low Power Computer Vision Challenge

Disasters like floods and earthquakes threaten human safety, infrastructure, and natural systems. Every year, disasters kill an average of 60,000 people, affect 200 million and cause $150 billion (USD) billion in damage. A timely and accurate understanding of the environment plays a key role in disaster preparedness, assessment, and response. Recently, unmanned aerial vehicles (UAV) with inexpensive sensors have emerged as a practical tool to collect situational imagery from disaster areas that can be hard-to-reach for humans. However, UAVs are equipped with energy-constrained supplies and low-compute devices, which limit the ability to perform automatic on-device analysis. This adds to on-board system latency, resulting in longer response times for disaster relief efforts. Therefore, achieving effective on-device computer vision with low power consumption and low latency remains a significant challenge.

In this project you will explore efficient implementations of semantic segmentation models collected from UAV and deployment of Jetson Nano.

Supervisor(s): Dr. Chaim Baskin and Moshe Kimhi

Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning/ computer vision and passion for efficient AI. This project will participance in LPCV competition as part of CVPR2023 conference.

Not just ReLU: Evaluate Activision on FPGA

Activision functions are a crucial part of deep neural networks, as a nonlinear transformation of the features.
In this project we will aim to burn an FPGA accelerator and measure the performance of different activations, including some that we developed here in the lab, to help the algorithm developers and see another perspective of choosing the architecture.

Supervisor(s): Prof. Avi Mendelson and Moshe Kimhi

Requirements: We are looking for someone with passion for deep learning and do not afraid of holding hardware board in their hands. Advantage: experience with FPGA + VHDL/Verilog

Quantizing Bert with new QAT

BERT [2], is the most popular transformer model.
Transformers, and Large Language models in particular, are very expensive to run and use.
One simple way to make this models faster it quantization [1].
In this project we will aim to quantize the Bert model by quantization aware training, with a new method called AMED[*], based on code provided with QBERT [3,4] and by the instructor.
[1] https://rasa.com/blog/compressing-bert-for-faster-prediction-2/
[2] https://arxiv.org/pdf/1810.04805v2.pdf
[3] https://arxiv.org/pdf/1909.05840.pdf
[4] https://github.com/sIncerass/QBERT
[*] paper and code will be provided after submission to the project.
Instructor: Moshe Kimhi and Dr. Chaim Baskin
Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning (e.g. 236781) and passion for efficient AI.

Supervisor(s): Dr. Chaim Baskin and Moshe Kimhi

Requirements: We are looking for someone with some experience writing DL algorithms. Elementary course in Deep Learning (e.g. 236781) and passion for efficient AI.