GSOC LibreHealth Automatic Labelling of radiology images

  " Experience " The experience that I have had in the past few months have been of a rather exclusive nature. The one that humanity has once every century, definitely not a critique on humanity or our definition of what is acceptable. I am just glad that I had got into GSOC and met the amazing folks at LibreHealth. The quarantine so far has been good and I am grateful for that. The learning was immense on many levels. The plannings we did to solve problems in our weekly meets to learning about new tech stack. It was all amazing. Definitely learned a lot more about Linux and git :). But I believe it is just the beginning of the journey and there is a long way to go in FOSS world. I plan to improve this project to the best of my abilities. I plan to keep in touch with my mentors and keep collaborating further, there are of course room for iterations and improvements. No software is ever complete. I just hope a lot of learning is to come in the future. Link to detailed project

  "Problem statement" Our app is geared towards improving the dataset in Radiology domain. We provide gui interface to visualise the inferences and modify the inference output. This helps in increasing and improving the dataset. To summarise ML domain in Andrej Karpathy's words "Data is king" and we are making the data even better. "Dataset"  The dataset that we worked on is NIH 14 Chest-xrays . The dataset has 112,000 images of over 30,000 distinct patients. But for Object detection it has annotation data for just 900 images. Which to train for medical purpose is just unusable. So we decided we will start with this dataset. "Web App flow"   [ Frontend ] Here is the flow of the app: Sign Up/ Sign In Get Image and start annotations: Things we can do: Select the Deep Learning model from which you want your annotations from Add more bounding boxes Remove bounding boxes Change the label of bounding boxes Add more labels to the label list Switc

  " Previously on LibreHealth GSOC 2020 "   [ Recap ] The app is ready, some requests were done by my mentors. which I have been adding. "Finalising features UI and everything"   [ Work ] Here is the look at the webapp in its full glory: This week I have added validation for diseases, and a logging on the backend for keeping the track of who is adding more labels to the already provided label. Tweaked the theme color to match the logo's color. I have also started the work on frontend segmentation tool. It should be finished soon. " Next week "   [ On LibreHealth ] 1) Complete the segmentation tool. 2) Start making final documentations.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The segmentation integration is completed and valid output is coming. "UI and other features"   [ Work ] 1) Added feature for downloading the annotation data in CSV format. Made all the necessary changes to the UI and Backend to facilitate this. 2) Added Snackbars for notifying user about the status of uploads. 3) Trained the retinanet few more steps to get better results. " Next week "   [ On LibreHealth ] 1) Start working on adding the segmentation functionality to the Frontend.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo backend and frontend was connected and they are working. Added support for UNET for segmentation. "UNET model served"   [ Work ] I have served the model and updated the frontend and backend accordingly. Everything on the DevOps side is working fine. But the segmentation output is coming out gibberish. I'am currently trying to figure out what is happening wrong. " Next week "   [ On LibreHealth ] 1) Complete the Segmentation integration. 2) Start looking into OHIF viewer.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo backend and frontend was connected and they are working. Added support for multiple models. Trained tf2 resnet. "Trained retinaNet in tf1"   [ Work ] The previous trained net was having too many bugs so I trained another model from scratch. Here are the diagnostics. I have finished the deployment part as well. and integrated React with Django. (Django serves the react app now). " Next week "   [ On LibreHealth ] 1) Add Image segmentation support to the app

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo backend and frontend was connected and they were working (with some bugs!). "Trained retinaNet"   [ Work ] This week has been all about training the retinanet model. Tensorflow just released Object detection API support for tensorflow v2. And it had bugs . So I ended up training the model from scratch and since there is a scarcity of data had to do some data augmentation to get nice results. The training went pretty well here is the loss curve: Although the loss looks pretty good, it can be further improved. And I will try that in the coming week. " Next week "   [ On LibreHealth ] 1) Start the segmentation part. 2) Improve this model.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo backend was connected to the frontend and some of the apis were modified for easy connection. "Frontend completed"   [ Work ] Requirements for frontend completed so fas: 1) Login/SignUp 2) User can request images with bounding box annotations. 3) User can filter bounding box annotations based upon which model was used for annotations. 4) User can modify the bounding box location/disease/size in real time. Here are few screenshots: " Next week "   [ On LibreHealth ] 1) Complete the training of retinanet and integrate it to the tensorflow serving server. 2) Make modifications to the frontend if requested by the mentors :)

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo of the app backend is completely dockerized and it is super easy to run now. "React js"   [ Work ] Now that the backend is ready it is time to work on the frontend. The frontend as of now me and my mentor decided will be in Reactjs. This week I made basic mods the react frontend added login page and sign up methods. Changed the backend to easily integrate with the React frontend. As I'm having placements I'm unable to work with my full dedication right now and I feel very bad about it. Anyways I will make sure I compensate everything by putting extra hours, all hours in the coming weeks. " Next week "   [ On LibreHealth ] 1. Complete the React app integration with the backend.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The demo of the app was ready but it was not dockerized. Running it would have been a nightmare. "Docker"   [ Work ] This week was all about learning about docker and make it easy to deploy my project. Which I successfully did thanks to docker compose. Docker compose is awesome if you want to build and run multiple container which communicate with each other. In our case as of now we have 3 docker containers communicating with each other. " Next week "   [ On LibreHealth ] 1. Add support for the react app.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The Django Backend and TF-Serving has been successfully implemented. The communication between the two is facilitated by Python based middleware. " Synchronous to Asynchronous "   [ Work ] This week was all about converting the python middleware into asynchronous service and generating Doc strings and documentation as the end of first month is nearing. Fortunately, I have completed my commitments for the first month and right no, I'm polishing the code. Python's Asyncio module was used to get the async await features. For facilitating dataset storage, I have looked into NextCloud. " Next week "   [ On LibreHealth ] 1) Train the NIH Xray dataset on a different object detection algo and add a support for selecting different ML algos. 2) Complete all documentations.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The Django server is implemented successfully. The Django backend is a Restful server that is responsible for communication between user and TF-Serving. Also, it is responsible for storing user data. "T ensorflow Serving "   [ Work ] The diagram above explains how we plan to deploy our ML models. Basically, the python middleware sends images to the TF Serving and receives BBox information and then send the received data to client. The client in this case is the Django Rest Server. The Django server stores Images and BBox data in a SQL DB. The User can now request for image and Bbox data from Django Server. As of now everything is implemented, but improvements will be made to the python based Middleware. " Next week "   [ On LibreHealth ] 1) Make the Python middleware Asynchronous and document everything. 2) Decide what type of file storage server we want to implement.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] The community bonding period has ended and we are clear with our requirements of what we are going to be doing for the summer. " Django backend "   [ Work ] The server in the above figure is a combination of 2 servers the django-server and Tensorflow servering. Basically, the Django server acts as the middleman between the Client and tensorflow serving. I have implemented the API for Django-server. I have used generic views and model serializers. The DB is sqlLite. " Next week "   [ On LibreHealth ] 1) Training the tf model. 2) Setup tf-serving container.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] There was change of plans, and the project's audience and essence was changed, for the better. Now the project has the potential to become something amazing. A new architecture was proposed to complement the new plan. And work on the UI was started. " The count down begins "   [ Final checks ] The coding is about to start and I'm ready. I am read up and very excited. The core client feature of bounding Box labelling is ready. And working like a charm. The frontend looks like this I know it is very basic but it will be improved, no need to worry. :) So, before the coding session starts we will be having a nice bug free UI to work with. Now I'm going to be reading all the articles I can find on tensorflow-serving. I wanna make this so robust and fast that it feels like a hot knife on butter. Even in React I have used only 1 class component, rest are functional. The UI needs to be polished and wicked fast.

" Previously on LibreHealth GSOC 2020 "   [ Recap ] After forking the repository and writing the Backend, it looked like every thing was going fine the papers were awesome and everything was going well. " The new Changes"     [ Change of plans ] The new project in its essence in is very different but technology and implementation wise it is not that big of a change. See I thought we were building a webapp for people to diagnose diseases. Well, that is not at all true. We are building a webapp that annotates diseases with bounding boxes to assist doctors during annotation period. To be a helping hand. " The new Architecture"      [ Scale ] The architecture will remain largely the same. But we are taking into account in a real world scenario multiple people will be doing annotations job so, it would be waste if we did inference for each doctor when they for images. We can speed it up. Here is what we will do. " The frontend"      [ BBoxes ] The fr

" Previously on LibreHealth GSOC 2020 "   [ Recap ] Kislay had celebrated his selection, told all about it to the friends and family, before he embarked on this journey, like a good traveller he has packed up, while in this context it was updating drivers, installing ubuntu and saying good bye to windows for some time. It was then he understood the quote " I get ideas about what's essential when I'm packing my suitcase" by Diane von Furstenberg. Now: " Django "   [ Gitlab ] I have created a new django app and setup a free smtp server from send in blue. The django app is capable of authenticating users by sending them email. I have committed all the changes to the forked repo. Since the plan is to use Reactjs frontend so I have used Django-rest-framework. For authentication I have used rest-auth. "awesome library" The steps to setup your own rest authentication service using Django: Install django rest auth and allauth: pip install django

" Automatic labeling of radiology images " [ Project ] I will be developing a stand alone web app. The web app will be able to segment, annotate and classify the uploaded X-ray through different deep learning algorithms such as GradCam and yoloV3. Now the architecture that I have proposed is a server client Restful architecture. The backend will be done in Django and Tensorflow serving( the machine learning framework of choice is Tensorflow2.1 ) and the frontend in Reactjs. The final goal is to make a beautiful web app that is scalable, responsive and beautiful. Here is a diagram of the architecture that we will be building: Project   [ Project architecture ] "Community bonding week 1" [ Work ] As first weeks at any organisation goes, I read up the forums and tried to get insights on the community. I also set up my development environment. I updated my visual studio code and cuda toolkit to 10.2. Enjoyed the fact that I got selected for GSOC 2020. "What can b