At ITU ARC, studies are conducted in advanced technology areas such as aviation guidance, navigation, control, unmanned aerial vehicle (UAV) technologies, air traffic control management systems and pilot decision support systems, as well as innovative materials and composite systems, wearable systems. As ITU News, we spoke with faculty member Dr. Emre Koyuncu and our doctoral students Mehmet Hasanzade and Omar Shaadeh about the project called “Deep Reinforcement Learning based Aggressive Collision Avoidance with Limited FOV under Dynamic Constraints” developed by ITU ARC.
Can you briefly talk about yourself and ITU ARC?
Emre Koyuncu: I am a faculty member at ITU Aircraft Engineering Department. This research center is a wide range of interdisciplinary research center, which we call the Aviation and Space Technologies Application and Research Center. I am the Director of the Control and Avionics Research Group of this center. In addition to the control and avionics group, we also have a research group in which new generation materials, especially for aviation and space, are studied. Only 60 doctoral and postgraduate students work in this building. Likewise, we have laboratories within the Faculty of Aeronautics and Astronautics. Around 20-30 doctorate and graduate students work there. In sum, in this center, we have advanced technology studies on artificial intelligence in aviation, guidance, navigation, control, unmanned aircraft technologies, air traffic control management systems, pilot decision support systems, innovative composite systems on the material side, wearable systems.
Can you give information about the artificial intelligence aircraft project you prepared?
Emre Koyuncu: As you know, the daily use of unmanned aerial vehicles has become widespread and we started to see these vehicles in the airspaces. In addition to the use of drones in the military field, we now see them more in direct cities for mail delivery or entertainment purposes. They became more accessible and their areas of use began to differ. In fact, they are now available in all areas. However, there is a basic control problem; decision making problem ... We can say that the control problem is partially solved. In other words, the ability of a UAV to do some movements in the air is no longer with a pilot sitting and commanding, but by giving commands autonomously. These studies will experience a transformation from classical control methods to aggressive control methods in a short time. We have been studying this topic for a long time in this research group. These will be reflected on the civilian side as well as on the military operations. For example, on the military side, we will start seeing drone to drone wars. We also see the newly developed kamikaze drone in our country; moreover, there are studies such as drones entering a certain region, collecting information and returning. On the civil side, for example, in film industry, there are a number of aggressive operational approaches to the creation of special visuals. As a research group, we have been working on their controls for quite some time. Generally, these tools are very small and the computing loads of the processors you can carry on these small systems are very low. Therefore, you cannot write very complex algorithms and run them. Now we have started to make these complex algorithms able to run with artificial intelligence. This project is actually a reflection of this. In other words, we run artificial intelligence on these systems - on very small systems - to be able to control very aggressive levels, namely avoiding obstacles, flying in a forest area, entering and exiting a cave, entering and exiting through a glass that it sees open, etc.
Can you explain the Unmanned Aerial Vehicle Platform you use in your project?
Omar Shaadeh: In this project, we did our test work on a quadrotor, which is basically a four-rotor platform. We did not want to use a large and high-cost platform for our tests at the first stage and apply our algorithms on it, because it would be a huge loss for us if something else happened. Therefore, we decided to use “CrazyFlie”, which is an open source, small and lightweight device developed by a Swedish company in 2016, which meets the needs of our project. We started to use the VICON System with it. We have eight VICON cameras at ITU ARC. Using the markers we added on “CrazyFlie”, we were able to determine the position of the drone with millimeter accuracy. Thus, the way we implemented our project turned into two layers. So one layer of the controller was working in "CrazyFlie" and this was the orientation determiner. This is basically the controller that controls the angles. Thanks to this controller, we can achieve the angle we want. We use position, speed and acceleration controller in the outer layer. In other words, the aggressive maneuver outputs produced were sent to "CrazyFlie" and these maneuvers were achieved very quickly.
What is Deep Reinforcement Learning (DRL)?
Emre Koyuncu: Deep Reinforcement Learning is a structure we use in our project. Deep learning has been developed recently especially with the participation of big players, it is actually a differentiated structure of artificial neural networks that we know since the 1960s. However, especially with the ability of deep neural networks to work very efficiently on the systems we call GPU, they have accelerated very much and these algorithms have reached a point where everyone in the world can access. I was also interested in this during my master's studies. In the past, you were writing these systems yourself, you were struggling, you were creating your own library, now ready-made libraries emerged. Then we started integrating them into the platforms. Where did it start? It first started with unmanned automotive. In other words, while the decision-making systems of unmanned vehicles use the methods we call rule-based, the work has begun to move towards the points where artificial intelligence is used. Vision and "perception" were very big problems. "What is that sign?", "What are the lights you see?", "Is it red or green?" It was difficult to perceive such details. Deep learning suddenly began to be a solution to these problems. These are complex systems, making decisions is not so easy. Especially if there are very complicated spots around you… It is easy to see the stop or late sign in white background, but it is a difficult problem to perceive that sign in complexity and to decide what it is. Artificial intelligence, deep learning, made these doable, workable and quickly applicable especially on real-time systems. Now it is at the center of attention of the world. “Deep Reinforcement Learning” is another point on top of it, it is a different optimization of algorithms combined with deep learning. They usually work offline, so you are doing some optimizations and offline learning about your system with Reinforcement Learning from the very beginning. Then, when you put it in the real environment, your systems start to behave the way they learn. You remember linear, non-linear systems, instead of deciding what to do with a linear modeling, imagine that all non-linear relationships are perceived and learned in the dynamics of the system. So you learn as a non-linear function and you are not very interested in what this non-linear function is. Deep artificial neural networks can keep these relationships within themselves. In this way, deep learning and Deep Reinforcement Learning - that is, the integration of optimization into deep learning - brought these systems to a point where they can now be done and work very quickly. We work at this point. Deep learning is a tool for us because we have different structures of Reinforcement Learning (RL), we work to make them more efficient. We carry out research on many subjects such as continuing learning in an offline environment and then learning in the real world, continuing the transfer of experience, aviation drones for aviation, and pilot’s decision-making systems in the cockpit.
What are the ways that algorithms follow in RL machine learning?
Mehmet Hasanzade: Our main question in this project was how an unmanned aerial vehicle can aggressively escape from these obstacles while flying and how to achieve it optimally. The Reinforcement Learning algorithm we use under normal conditions is a system that knows nothing, really like a baby. With the commands given by his family when he tries something, this baby is said to be doing right or wrong, and now he starts learning at a certain point. The logic behind the Reinforcement Learning we use works the same way. The system does not know anything, but there are actions it can take, such as changing its speed and angle. Trying these different actions and saying that the actions are always right or wrong, the system has learned to fly better. We want him to avoid an obstacle, but our vehicle has a size. Therefore, we set a certain safe distance. We determine the distance and we do a lot of tests, always telling him to escape at this distance. These tests are normally done in a computer environment. The same medium is still a computer medium. We construct the same environment in simulation in a computer environment and then perform many tests. I mean by many tests, millions of tests. In these tests, show him the right and wrong, our vehicle starts to produce an optimal escape maneuver without hitting the obstacles. In our project and in the video we publish, we realize that our vehicle can escape in a complex environment with optimal maneuvers without hitting these obstacles.
Will we see this project you prepared for IROS 2020 in our lives in the coming years? Can you talk about your social contribution?
Emre Koyuncu: In IROS 2020, the largest robotic conference in the world, you can see quite complex decision making, control and artificial intelligence problems related to ground robots and water robots. The aviation field is a bit more difficult. Both operations are risky and there is more complexity. When we look especially in terms of state space, we are working on systems with a large space and more complexity in this respect. Especially when you solve the problems in this complexity, you actually offer solutions to the world about the problem. For example, let's talk about current planes. There are collision avoidance systems, which we call TCAS. These are generally driven by geometric algorithms and rule-based choices rather than optimization in themselves. If I am faster than him and if I am a little up, I am climbing or I am under him, if he is coming towards me, I will keep my speed and wait for him to pass ... These are the systems directly on the aircraft. Such self-learning, self-improving systems can now be integrated directly into these structures. In fact, we are creating a very safe world for large aircraft with these systems. They can parse when they see each other from 3 miles away. Now the world is already very safe, there are almost uniform aircraft, and there are performance models with very small differences in themselves. However, imagine that unmanned aerial vehicles are involved, that they start flying in cities, and that they work as taxis that take off from airports and take you to cities. We are not talking about a very distant world, after 4-5 years we will start to see them. We're talking to Boeing here to do the first trials. Different countries such as Brazil also started this. While I was at the Massachusetts Institute of Technology (MIT) during my doctorate education, researchers from Amazon made a presentation that we would send mail with drones. They told us how to do it, but we did not believe it. We asked questions such as; "Can such a thing be possible?", "How far can you fly?", "How big can your fleet be?" . 4-5 years later it became reality… In fact, these developments are not so slow and we will come to the places where unmanned aerial vehicles interact with each other after 4-5 years. At that time, it will not be possible to solve a system of this complexity with rule-based structures, algorithms that we call intuitive. In this case, it will have to be self-learning, decision-making systems, artificial intelligence systems that can solve their own complexity. We will use these structures here, and as I said, aircraft contain complex problems, when you solve this problem, you offer solutions for various systems on the ground.
What is the focus of your recent work? Could you give some information about the other 3 projects supported under the European Union H2020-SESAR-ER Program?
Emre Koyuncu: In this research center, we have a lot of work on unmanned aerial vehicles. We are working on different drones that can fly for particularly aggressive defense industry in Turkey. We have partners such as Aselsan and Havelsan. We are starting a big project with our defense industry company with the aim of developing a large unmanned aerial vehicle platform. We will launch the project soon. In addition, we have important partners in European Union projects. For example; While working with Boeing on complex non-linear equations of performance models, we work on points such as self-learning of the parameters that we know to be uncertain when the aircraft is in the air and making calculations related to that flight route even more sharply. Our 3 European Union projects are starting. One of them is our project, which we call FACT. On flight systems where new generation avionics called GNSS, GNC, and systems like the new 5G will be used. We will work on these systems in general aviation, helicopter, unmanned aerial vehicle. Using these systems in operations by integrating them, they will begin to localize themselves not only by using satellites but also through base stations when they approach urban points. In this way, it is possible to run algorithms with higher complexities, escape from buildings or start fields that we do not want to enter. Players will be very different in the airspace. We will not only talk about large commercial aircraft. At the same time, small unmanned aerial vehicles will have to be able to work together in the same airspace, each of the systems controlled by people and used for transportation such as a taxi. Here, we will work on them in our new European Union project. Our other European Union projects are also related to the integration of airspace. We will work on developing performance models of aircraft, seeing the air transportation network as a whole, and perceiving a problem that can be encountered in the airspace again using artificial intelligence and making some operational decisions accordingly. Another one is what we call ClimOp, how we can keep up with the climatic changes occurring in larger time scales… We are working on the entire model and simulation of European airspace at ITU. Our partners will focus on what changes in operations related to climate change. We will perform the simulation, analysis and optimization of this whole system here at ITU. In addition to these studies, we also have research in areas such as evaluation, analysis and optimization of our airspaces.
So conventional practices will change…
Emre Koyuncu: We definitely see that these should change in terms of security. Maybe we do not see a lot of aviation accidents, but the points very close to the accident immediately enter the radar of the researchers. We have a constantly active and dynamic field of work, such as why it occurred, what its causes were, what kind of problems were experienced in pilots with training, what we should do about it, and what technical problems there are in the aircraft, how we should correct them. We keep the issue of security at the top of everything and as this center and working group, we are at the very center of it.
About Dr. Emre Koyuncu
Emre Koyuncu is a faculty member at Istanbul Technical University, Department of Aircraft Engineering. He received his Ph.D. degree from ITU Aircraft and Space Engineering Department in 2015 as a SESAR JU PhD Fellow, and as a guest researcher at Boeing Research and Technology of Europe in 2013-2014 and at Massachusetts Institute of Technology (MIT) in 2014-2015. His research focuses on artificial intelligence / optimization-based control and probability theory applications in high performance cyber-physical systems. The scope of application areas covers; high performance aircraft, flight route planning and optimization, obstacle avoidance and separation systems, high level autonomy in air traffic control systems, air traffic network models, flight management and cockpit decision support systems. Koyuncu has managed and conducted many research and industrial projects with stakeholders such as H2020, SESAR, BOEING, ASELSAN, TAI, TÜBİTAK BİLGEM, STM, Turkish Airlines and IGA. Koyuncu, who has more than 50 published articles, was awarded the Boeing Career Award in 2015. Approximately 20 doctoral and graduate students work in the research group of Koyuncu, who is a member of the Board of ITU Aviation Institute and Director of the Control and Avionics Research Group at ITU Aviation and Space Technologies Application and Research Center (ITU ARC). Koyuncu is also the Associate Editor in the IEEE Access Journal and a technical committee member of the IEEE Control Systems Society on Hybrid Systems.