DevOps Chats: The Future of Farming with Aerobotics

Spinnaker Summit 2019 Preview: Never underestimate an entrepreneur looking to solve a real-world problem. Aerobotics is bringing drones, imagery, mapping, containers, Kubernetes, Spinnaker, data pipeline processing and machine learning to agriculture. Sounds interesting, right?

Aerobotics uses aerial drones and sophisticated image processing to help growers manage yield, pest and disease in their farms through artificial intelligence.

Speaking at the upcoming Spinnaker Summit in San Diego, Aerobotics’ Head of Software Nick Coles, shares with us how his company evolved from making drones to map orchards into a software company using raw image data to create high resolution, georeferenced maps of orchards. Aerobotics technology is comprised of a map engine and tree engine. It’s both a fascinating problem space and exciting application of cloud-native technologies, including Spinnaker.

This episode of DevOps Chats features a preview of Nick’s talk, “The Future of Farming with Aerobotics.” Nick’s talk is on Sunday, November 17 12:30 PM, at Spinnaker Summit 2019 in San Diego.

 

https://api.soundcloud.com/tracks/701186533

Transcript

Mitch Ashley: Hi everyone. This is Mitch Ashley with DevOps.com, and you’re listening to another DevOps Chat podcast. Today I’m joined by Nick Coles, head of software at Aerobotics. This is a special edition of DevOps Chat, because we’re talking about talks at the Spinnaker Summit 2019, which is in San Diego this year.

Our next topic is the future of farming with Aerobotics, aka drones. So you’re gonna be really interested to hear about this. This talk is on Sunday, November 17th at 12:30 p.m. Nick, welcome to DevOps Chat.

Nick Coles: Hey Mitch. Thanks for having me. Really excited to be on this podcast.

Ashley: Happy to have you here. Tell us a little bit about yourself. Tell us about what you do at Aerobotics and a little bit about the company.

Coles: Cool, so I guess just from the very beginning, born and raised in Cape Town, South Africa, as you can probably hear by the slightly strange accent. [Laughs]

Ashley: Detected. We detected a slight accent there. [Laughter]

Coles: Yeah. And but yeah, born and raised Cape Town, South Africa. Studied at the University of Cape Town, where I did a electromechanical engineering degree, which was kind of, you know, embedded systems, microcontrollers, processes and stuff. And that’s kind of why I got interested in drones and robotics. And, yeah, just over four years ago joined Aerobotics and sort of–at the time it had just been funded, so there were essentially two people, and I kind of joined in a software engineering role, not really knowing what I was doing.

Ashley: [Laughs]

Coles: And yeah, kind of been there ever since heading up the software engineering team. Which obviously a huge component of it has been DevOps and back-end processes. And so that’s kind of why I sort of have been exposed to Spinnaker and, yeah, kind of what led me to this today.

Ashley: Excellent. So it’s the true startup experience, which means your job is you do whatever needs to get done, and you kind of evolve into your current role, it sounds like.

Coles: Yeah, pretty much. I mean I joined–back in about four years ago when we joined it’s basically we were a company that was building drones, building the hardware, sort of soldering the motors to the propellers. And essentially drones kind of transformed throughout our journey, and we ended up having to move out of the hardware space and peel into the software space, because companies like DJI came up with these commoditized perfect drones. And it was quite hard to compete with them when you’re, what, five engineers in a garage soldering drones on a piece of [Crosstalk]–

Ashley: There are a lot of drones available and you can differentiate in the software. So I read from the description of your talk that Aerobotics uses aerial drone imagery to help farmers, growers, manage their yield, pest, disease, that kind of stuff on farms, using some AI. Is that a problem that you started initially with, or did the company evolve to that kind of problem? How did you get started, and the company get started around that area, specifically?

Coles: So, I mean, the company has actually had quite an interesting sort of journey in that regard. So the idea in the beginning was we are drones. We build drones. We use drones for agriculture. We use drones for security, game counting, mining. We tried to essentially just throw drones at any sort of problem.

And sort of as we progressed as a company and more and more players sort of entering the field, and we sort of started honing in on our value proposition, and started specifically focusing on tree crops within the agriculture space because it’s such a high value. And so it actually then makes sense for the cost for specific tree crops.

But originally we weren’t using any AI. We were kind of just trying to use raw sort of computer vision and object detection. The data that we were seeing is so vast, and case-by-case is so unique, that we kept having to change the parameters, slightly tweak things. And so the only sort of long-term solution could’ve been machine learning. And so that’s kind of what we embarked on about two to three years ago and sort of been, yeah, what we’ve been using ever since to try to essentially generalize the imagery we were receiving and sort of try to replicate the human behavior to the best of its ability.

Ashley: Okay, excellent. So I know your software staff appreciate that about the company in kind of the broadened space. I know your software stack is a combination of, what, Kubernetes, of course you’re using Spinnaker, I think Circle CI for your CI/CD. Tell us a little bit about how you got involved with Spinnaker initially. Was that something that you chose or kind of evolved there kind of by accident? How did you happen down that path?

Coles: So, I mean, it is–it did happen by complete accident. Traditionally when we were processing all this data we were just spinning up EC2 instances. And so we were essentially just sending SQS messages and spinning up instances. And instances would essentially be able to sort of, I don’t know, kind of retrieve these messages and then figure out what sort of processing to do. You know, it was an absolutely terrible solution for instance _____ a drone with a code. We’d have to SSH into the instance, patch it, figure out what was going on.

Ashley: Oh boy.

Coles: At this time also it wasn’t–like, DACA hadn’t really sort of like exploded. And or maybe it had in sort of like San Francisco and everything. But in Cape Town we weren’t really exposed to all these sort of advanced kind of tech solutions. So we had quite an archaic little approach going on, which was working, I guess, with R Scale.

And then in the beginning of last year we got accepted into Google Launchpad, which was really exciting. Four of us went across to San Francisco, and we basically went on this launchpad program where they kind of helped us all the way from technical to business. And it really was inspiring to kind of go to San Francisco for the first time and sort of see like just the scale of things and just like the intensity of the tech companies there and really what they were doing.

And we kind of got put in touch with a couple of mentors. And one of them, based on sort of what the problems we were trying to solve were and the problems that we were facing, recommended Spinnaker. And at that time we were kind of on AWS’ stack. And so we kind of then got incentivized to sort of try, set up a system on Google. And you know, we had like Spinnaker set up, Google’s Kubernetes’ engine, and kind of we just took it from there.

And just it was an interesting journey, ‘cause there was quite a lot of iteration within like Spinnaker itself and in terms of just how the community’s progressed over the last sort of two years, but also just sort of like where we were at in terms of not really understanding how Spinnaker worked, the containers spiraling out of control, realizing we needed to stabilize our Redis pods. Like our Redis pod basically caused so many problems early on.

But yeah, so that kind of was the journey in terms of how we got involved in Spinnaker. So it was basically just by chance somebody mentioned it, and we kind of were like, wow, this shows the problem we’ve been having.

Ashley: Like a great solution.

Coles: And we kind of just ended up, yeah, just going with it.

Ashley: Fantastic. So from what I understand is the drones that–just giving our audience a little bit of context–the drones that are flying around the orchards, for example, are collecting all of this data. And that goes up into some data processing in the cloud, correct? And it does, I guess, splits into two areas. One is a map engine that creates–takes that raw imagery and creates multilayered geomaps of the orchard. And then the tree–a tree engine, which also then looks at specifically where exactly are the trees. And that’s how you do your processing. Is that a correct description of how this works at a high level?

Coles: So it is basically quite a good description. So just to maybe give like slightly more detail on like the map engine side, we are–I don’t know if you mentioned–but we are using multispectral cameras attached to the drones. And from that data we can get information that isn’t necessarily visible to the human eye. And that kind of gives us all a health and powerful readings from the plots.

Ashley: Is that like using infrared kind of things, or what?

Coles: Exactly. Exactly. Infrared sensors. And so, yeah, so the map engine side is essentially like the big data processing side of things where we’re taking thousands of thousands of images and stitching them into these high resolution maps of individual orchards. I mean, just to give sort of some context on kind of what the data is there, we generally produce about like these five gigabyte high resolution images, which is of a specific orchard. And the kind of process around getting the raw imagery to that five gigabyte sort of image, we use about 64 gig RAM and 16 CPU instances. So that’s kind of just like the scale of what you’re running in. It can take anywhere between two hours to 24 hours to form that stitching process.

And then moving onto the tree engine, this is sort of where our machine learning models are running in production. And that’s essentially taking these high resolution maps and segmenting the image to get individual trees from that image. And from that we can then calculate additional metrics, such as the height, the health, and the area of the individual trees.

And one thing that, I guess, we left out because we are still developing it, is we now have a fruit engine, which is essentially now like kind of our new product focus area where we’re going from the drone, which is kind of taking a high level of the orchard, and now we’re going right up to the individual trees. And from those trees we’re starting to detect fruits and sizes from the fruits. And the goal here is to give the farmers an accurate sort of size distribution of the fruit in the orchard so they can plan accordingly when it comes to harvest time.

Ashley: They’re all–it’s all about yield, of course, and quality of the yield as well. And by the way, I come from–I’m not a farmer, but I grew up in the middle of Nebraska in the U.S., so I’m around a lot of farmers. So I have some idea of what you’re talking about, at least an appreciation for it.

Well how did–tell us a little bit about why you chose to do this talk at Spinnaker Summit and some of the things that you want to communicate beyond kind of the problem set or the solution that you’ve come up with.

Coles: Okay, so one of the reasons why I thought it would be quite cool to speak at the summit was, I guess, just based off like chatting to some of the Spinnaker guys on the Slack channels and everything, it feels like we’re using it in quite like a unique sort of way. I mean, we’re using it very much to manage essentially our data pipelines. And I guess we’re almost using it in a similar way to how someone may use something like Apache Airflow or something.

And I thought that the combination of the uniqueness of how we’re sort of using it, and sort of just to kind of explain how it has completely changed the way we’ve been able to develop and iterate super quickly, I thought it’d be worth kind of sharing that with the community and sort of seeing how other people were using it in comparison. And so I guess, yeah, that was kind of just the main thing, just the type of data that we’re sending through the system is quite different. It’s quite unique, in comparison to the rest of the world.

You know, we process these large images which come up with a–which have a whole bunch of different problems with them. You know, we have to ensure that our containers are responsible, they’re not using too much RAM, they’re not spiking in terms of, you know, being out of memory because they’re loading an image all at once. And there’s just been a lot of interesting things that we’ve had to sort of develop to kind of handle our case. So I thought it’d be cool to share that with the community and take it from there.

Ashley: You know, I think it’s super relevant, and not everyone may have an agricultural problem domain, but with a massive quantity of data that we’re collecting through sensors, whether it be drones or IoT and home or IoT and the business or any device, any systems that are collecting all this data, one of the real challenges is how do you process this much information? How do you combine it together? How do you take different sources like your map engine versus the tree engine, look at it, and then do that analysis and produce valuable, interesting results?

I think a lot of businesses, whether they be startups or enterprises, have those kinds of projects and challenges, or at least they’re thinking about it, if they’re not already doing that. And I bet you’ve got some valuable lessons that you’ve learned over time about why you set up the pipelines the way you did or how to handle some of the production challenges.

Coles: Yeah, you know, it’s also we had an interesting space, also, because we’ve learned so much over the course of the last two years. Like when you’re doing things in production, you also very quickly start figuring out what it should be. And so there’s even more and more of an end-goal, if that makes sense.

Ashley: Mm-hmm.

Coles: So like as a small example, kind of, you know, we’ve set up all of our Spinnaker pipelines, and it’s been amazing. And the devs have been able to iterate really fast. And now we’re starting to see that we want to maybe start introducing some stability to our pipelines. And so to achieve that you want to start setting them up as infrastructure code. And like just starting to build that whole process into the development cycle is obviously quite challenging and takes time. But it’s allowed us to sort of think forward and continually improve on what we’ve been doing.

So you always feel like you have kind of found the best solution at a point in time. You’re like, wow, this is absolutely incredible, and it’s going so well. And then six months from then, you’re kind of like, wow, it could be so much better. There’s so much more we can still keep doing to improve again. So I guess that’s what’s been quite an awesome part about using Spinnaker. It’s like at every sort of milestones it’s kind of like you can keep iterating and making it better and using it more efficiently.

Ashley: That’s excellent. When you use a tool like Spinnaker open source to a software like that, that you feel like you’re just getting started sometimes. You thought you knew what you could do with it, and you find out there’s eight or ten different things that it evolves to from where you started.

Coles: A hundred percent.

Ashley: Well good. I wish you the best of luck on your talk. You know, I think you may get some folks attracted just to come hear about the drones. I hope you’re gonna have some pictures of maybe some of the orchards and the maps and some of the output that you do, is that correct, just to give folks an idea of kind of how this works?

Coles: Yeah, we’ll definitely, I think, do a very brief sort of couple minutes on the product and just kind of explaining what we’re trying to do, and for sure. ‘Cause I guess it’s always nice to sort of give context of what Spinnaker is actually producing at the end of the day.

Ashley: Mm-hmm. Well very good. Well it’s been great talking with you, and I wish you the best with the talk. And hopefully you get a lot of folks coming that are excited to hear about what you’re doing. Is this the first talk that you’ve given at a conference like this, or you’ve done that before?

Coles: No, this would be the first talk, so I’m quite excited for that.

Ashley: Really? Excellent. Well you know it’s–there’s lots of ways to give back to an open source community, and of course, writing code is certainly one of ‘em. But most people don’t write code. And I think one of the great ways of giving back is doing exactly what you’re doing, is going and talking, not just in the community and online forums and Slack channels or whatever they are–that, too–but participating in talks like this.

Because it–I’ll predict you’ll have a lot of folks come up to you after the talk and want to know more or talk about what they’re doing and tell you about their idea or ask your advice on something. So it’s a great way that advances the software and the use of it, because you’re doing things that probably other people never expected or predicted. So congrats. That’s awesome you’re doing it.

Coles: Thank you. Yeah, I guess that’s one of the things I’m most looking forward to is just sort of meeting a whole bunch of new people and kind of, I guess, like we learn so much, like especially kind of being in South Africa, not kind of in the tech hub of the world, I would guess, is when you go there and you start speaking to some people who are kind of leading the way in terms of the industry, it’s so valuable just to kind of hear what they have to say and what they’re using and how they’re solving problems. Because it’s so relevant and it can translate so nicely back home. And so, yeah, it’s always so much value in just meeting the people and just chatting through things.

Ashley: Well excellent. Nick, thank you for joining the podcast today.

Coles: Thanks Mitch. Will you be at the summit in a month’s time?

Ashley: I don’t think I’m gonna be able to be there. I’ve had several invitations [Laughs]. If I can I will be.

Coles: [Laughs] Okay.

Ashley: But I’m like back-to-back for the next month going to conferences. So we’ll see. Hopefully. If I make it I will definitely look you up.

So I’d like to thank Nick Coles, head of software for Aerobotics, joining us today. Again, he’s speaking at the Spinnaker Summit 2019, the dates are November 15th to the 19th in San Diego. By the way, Nick, you’ll enjoy San Diego, too. It’s a great city like San Francisco is. His topic is the future of farming with Aerobotics, aka drones. And on Sunday November 17th at 12:30 p.m. So please check out his talk.

And thanks to all you for joining us today on this podcast and listening to Nick and I chat about his talk. This is Mitch Ashley with DevOps.com, and you’ve listened to another DevOps Chat. Be careful out there.