I'm not convinced that Android multi-module projects are the way to go anymore. I'm finding gradle indexing, building and rebuilding is taking up the majority of my day now. It feels like I'm barely coding anymore and just waiting for the tools to catch up.

I appreciate the hardwork that is put into these tools, but at this point I'm starting to hate developing for Android with large projects. With small projects its a breeze.

The solution? I think it might be putting modules into a maven repo and publishing artifacts. Then when you need them, pull in the artifacts like you do with AppCompat, etc. Use what you need. If it changes, update the version code, reimport. All done.

There's simply too much swapping, building, indexing, and time wasting going on. I want to build software, not fight the software that I use to build software. I'm not alone in this, I've spoken to hundreds, if not thousands of other devs who feel this way.

I don't disagree but I can tell you that in a lot of cases people are just going to get into dependency hell where they will need to update 3 dependent libs to change one thing. Why? Because finding good boundaries for libs is what mostly doesn't workout.

Agreed man. However, when this happens I can still build quickly. Thats my concern here. But you're right, there are other issues, but IMO, not as bad as this one.

You're right, maybe sometimes less code is the solution not just trying to push it further away into seemingly immutable packages. I've left Android for Go so I'm not sure how much there is in a typical app now, but back in my days it was already too much and crazy 😂

I've witnessed a number of developers leave Android for something else. I think we'll continue to see that happen as its just frustrating at times.

Have you ever worked on Android project without annotation processors? If yes, did you find this approach less frustrating?

Far less frustrating. Smaller the project without annotation processors, the better.

Have been thinking for a while about trying to avoid annotation processors after seeing build stats of client's app. It's a pile of crappy code, but there were no annotation processors there, and it builded quite fast.

It makes sense if you think about it. No need to codegen every build. Writing files takes time, finding dependency graphs and resolving them takes time, etc. Eliminate that, and poof, problem solved.

I wish Glide never implemented na annotation processor. Really no need for it. DI I wouldn't mind doing manually, but not image loading... Might be time to try Coil!

Or @kodein. Wonder if using that is why I don't have many issues with build times and multimodule. Guess my project isn't super big either. Self enforcing architecture mechanics or fast build times seems a tough trade.

Are you able to elaborate on where these processors were used? e.g. One giant app module, a module with many inward dependencies or a module with many outward dependencies?

That app didn't use any annotation processors at all. It was about 65 KLOC of pretty bad code (MainActivity and MainFragment 5 KLOC each), but incremental build was ~10 sec om my MPB 2015.

And just two modules. The second one was basically imported open source lib that they modified.

Oops, I missed the word 'no' there

Gradle as a build system fundamentally fails as soon as you break something like incremental builds (*cough* annotation processing). Bringing annotation processing into your builds is a trade off - reduced code complexity for increased build speeds.

I would blame the fact that Android Studio and Gradle work with radically different project models. These problems will persist until all workflows (build, debug, and test) genuinely rely on a unified engine.

They are right to not light the status quo, but it's not caused by multimodules per se, but by a bad implementation of it

Keep the baby and throw away the dirty water Ask the Android team to remove their stupid Android build variant. Ask the Kotlin Gradle plug-in to use the lazy Gradle API Ask jetbrains to fix https ://youtrack.jetbrains.com/issue/KT-32158

Also relying less on annotation processors helps a lot. Kill dagger and replace it with Koin

I agree. The build system is difficult so I empathize with those people but it's crazy how lots of standard libs are used in Android that completely kill productivity. I started doing react dev. Wow. Is that a breath of fresh air. I get more done in 1 day vs 1 week in android

This is why I’m excited for the github package registry stuff. It should make this workflow much easier.

Worked at one place that went down this path, including an attempt to set up proprietary Artifactory. IIRC, wasn't that good of an experience either, but I'm not sure whether the infrastructure there was any good (was inexperienced myself).

This is exactly what I did for @powerley when I was Lead there for the Android side of things. That plus Gradle opts plus mainframer = 😍😍

Have you tried Gradle composite builds? It might help abstracting the modules that are more SDK/Library style.

I have not, thanks for the link

During my time at my old company we started doing this and published an artifact for each feature. So each individual feature was an artifact and we had an umbrella app that combined all the artifacts. At my current company we have a similar approach. Works quite nice tbh 🤔

In this approach, if a fix needs a change in multiple modules, then it is much difficult to work! You need to have all the modules imported in Android Studio to do continuous change and try the fix. 😞

Not if you split everything smart and do not couple things too much 🤔

Define "smart"

Slice everything per responsibility and treat everything as its own library. You will not get around on touching more than one project some time, but as long as your public apis do not change often it should be enough to release a new artifact and build the umbrella with new ver

Sure if a new feature needs a new API it might change more then one artifact, but usually if you slice in a way that every module has a concise API you should mainly be changing internals for bugfixes and can release independent from each other.

Maybe I was also lucky with the projects I did till now, because they made it easy for me to find good slices. But what we did was kinda like: Umbrella Feature A. - Feature B. - Feature C - REST for A

What also helps is writing tests and having a nice CI/CD system

But yeah really hard to describe the thought process itself on Twitter, maybe I get to writing some thoughts down later this week

The problem is not multi module, is gradle. At scale, publish all modules as artifacts and do versioning is not scaling very well. Bazel or buck (or a fixed-if-ever Gradle) would be the solution.

Bazel or Buck have no dependency management and no fine grained incremental builds whatsoever. That's no surprise OkBuck uses Gradle builds to generate Buck build files, it would be a nightmare to handle otherwise!

Not sure that's true. We use it internally very successfully at scale.

I maintain github.com/bazelbuild/rul…, which provides external dependency management for Bazel through Coursier. It's not perfect, but it works for most use cases. There are a few others too.

I admit that with large projects, running android app takes a lot of time, but comparing to typical architecture, modular architecture really minimise building and compiling time, and that doesn't mean that we are fully satisfied about that speed.

We are facing similar scale issue with build system in a multi modular setup of our app , wait somewhere between 12-20 mins , many optimizations in , we have come to about 2 minsnavg on incremental builds , ci takes roughly 35 mins

I can't stand it. It's making me hate developing on Android.

I feel the pain , simple feature testing becomes difficult , wait and then more waiting till build is ready

Then realize you didnt do it right, change one line. Wait ... wait ... wait ...

Circle of android dev life , coffee gradle kotlin gradle ...Then some coffee , gradle .. frustrating (got close to 100+ modules)

We actually have started to track our developers local build time and average out where the max configuration indexing time bottles out , but for now came up with this “Android Studio — Build Time hack” by Akhil Gupta link.medium.com/oLHZ8UkbQ0

Paradoxically we all would expect Gradle caches to make multi module projects compile and index considerably less than single module ones. Maybe it's us not splitting the way that helps compiler (high cohesion) but the way that fits better our sense of architecture.

This very well could be part of the problem. Therein lies another problem - having to know the intricacies of a build system so that you don't shoot yourself in the foot. I want to build software, not configure a compiler. I think you're onto something though ...

I think it's good to have a sense on how tools work under the hood, but indeed imho it should never be a stone in the way. I dislike the need to organize a project in a way that satisfies the compiler instead of the team organizational needs.

Ditto man, ditto. 100%

'I want to build software, not configure a compiler'. I couldn't have said it any better.

🙏🏻

Gradle is like Schröndiger's cat: - Gradle is awesome - "Gradle" is terrible - both are true at the same time. Try gradle on the backend and you will like it What sucks is the all the Android build tools, aka "Gradle"

I don't doubt that. Gradle is way better than the predecessors, no doubt.

The problem is bad documentation. I have written a series of articles to try to overcome this, have a look there:

Well @aditlal did some work on this to drastically bring down their build times by structuring the multi-module project with minimal inter dependencies. It does take a bit of planning beforehand. He did a talk about it

I’ve had this exact same conversation with a friend about Scala/sbt. Honestly do not want to spend significant amounts of time waiting for a build system especially after seeing a good one like OCaml’s dune do near-instant recompilation and get faster with each release.

How do you decide what to modularize? If you create a module, do you let it depend on other modules? Should it? We've been having success with flat dependency trees, and bridging adapters.

For example if you have a screen that shows data from the web: create a module for the UI and one for retrieving the data. They're completely orthogonal and depend on zero modules. A third module depends on both and contains adapters.

This is "ports and adapters" or "hexagonal architecture" by the way

This is a very hard question to answer and one I unfortunately don't have a great answer to ... yet.

Yup.

There are different issues here: AGP is quite terrible, if you use Jetifier it's even worst. Kotlin doesn't support compile avoidance, so any implementation change forces recompiling too much, not like what happens with Java. And modularizing in the right way is not easy amyway

That's quite revealing. Can you share some good literature about both issues?

For Kotlin and Compile Avoidance I would start from: github.com/gradle/gradle/… For Jetifier..I think it will be properly cacheable starting on AGP 3.6.0, they use Transforms and it's not even shown properly on Gradle Scans ( will be soon )

Thanks! I'll take a deep dive on both 👌

I previously worked on a project with 20+ modules and I could spend half my day building the project after making a gradle change. Such a waste of time.

Do you have an idea after how many lines of the code it starts to get slower?

I dont, I have not kept track

Agree on the assessment, not on the conclusion :) Gradle is a mess, and in fact Android Gradle Plugin more than anything else. But the virtue of splitting into modules helps blocking you from introducing nasty cycling or unexpected dependencies. 1/2

Pure Java modules helps but the tooling should definitely improve. It's a shame Gradle build scan exists. Every Gradle release: we improved speed YAY, \o/ Still, it's really painful to work with AS & Gradle (+ you don't want to bring C++ external native build on the table) 2/2

What is the depth of your dependency tree? Would flattening the tree help?

Have you ever been to the redwood forest? It's big. Will/would it help? Maybe. Who knows. Thats a project in itself ... 😂

We've got a 30kloc project with three roots, 52 modules, max 5 nodes deep. I like to think it helps 😅

TBH, I wouldn't bother modularizing 30KLOC project at all for the sake of build times. 52 modules? Doesn't it make navigating and maintaining the codebase a bit harder?

It's modularized like that because of code reuse between 3 apps.

Also the modules pretty well defined, the folder structure invites you to find what you need pretty easily.

Especially modularization helps in maintainability because concerns are separated by default. There's no way web services or repositories are referenced in UI modules because it just won't compile.

See this tweet for a coarse example: twitter.com/n_haarman/stat…

No arguing that. Modularization helps with structure and organizing. Build times ... Ehhhh "it depends"

Agreed 30k lines isn't that much

at Capital One, we use a testharness to run,test and, debug each module/project separately without requiring to sync the entire project every time.

That definitely helps for some parts of dev, but that's only because other dependencies are published as binary builds. I'd love to have an out of the box solution from Android and not have all the setup/work that is needed now.

This.

Ultimately the issue is that good Gradle practices are opt-in whereas Buck/Bazel force you to opt-out of the default good practices.

The "Test Harness" is somewhat similar to what @yogurtearl mentioned as a "dev build" flavor, where you just have the bare minimum of what you need.

👋

Have you tried dagger-reflect? Got some good results with this. I agree though, for me extra modules are not worth the overhead.

isn't android development a ping pong? back to reflection?

Dagger-reflect is only supposed to be used for debug builds though, so not really

You can right click and "unload" a module from Android Studio to avoid indexing cost.

Well, It is IntelliJ feature actually not even AS specific :) blog.jetbrains.com/idea/2017/06/i…

Why doesn't AS get these features in? Second thought, thinking of moving to IntelliJ 🤔

AS is based on IntelliJ, so from the previous comment I understand it also have this feature. Thanks @yahyabayramoglu

Should I do that with all 100+ modules? 🙂 Probably not feasible...

Yeah, I kinda expect it to be reversed as in you "load" what you need rather than "unload" what you dont :)

Which goes into direction how @BuckBuildTool handles that problem

What's your incremental build time in a (not-so-small) project with a single module? Thanks to having more modules, we keep incremental build time ~15secs on average and ~2secs when you run tests. These are numbers w/ kapt enabled.

I would never know but if we had a single module, build would be up to at least 1 minute.

1 minute would be a dream

I mean I've seen non-modularized big projects have 3-4 minutes incremental builds. So I am definitely in for modularization even with additional indexing etc.

It's an assumption at this point. Meaning it's still moduarized but the "modules" are their own projects that are pulled in via artifacts. Just a theory at this point.

This is a slippery slope to @BuckBuildTool and @bazelbuild

Flutter

Perhaps, this depends on the way you structure your module and in which module your making changes. We are enhancing our modular project by splitting app module to individual feature modules and putting lots of thoughts on how we gonna structure it.

(No comment)

I think its a tooling problem... is it that bad on our side?

Naa it really isn't Sha.. Initial build about 5-7 minutes.. Subsequent builds 30secs-2mins.. Unless you change something in core and then you're back to 2-4min builds

thread

Completely agree but Composite Builds will help alleviate some of the pain.

Its all because of gradle build system ( afaik ) which is not memory efficient and too slow. Trying out bazel build system can help

Try to switch branches with Android Studio it will save you from reindexing and etc.

Can you share more in this? Why would this matter?

If you change branch in AS it will load where you left and omit indexing. I found it much faster when for example switching back to dev or feature branch I worked before.

You had to do something wrong. We were trying to split app to more then 30 modules to speed up our builds and now this is perfect. Make sure you got all needed flags like "kotlin.parallel.tasks.in.project". Also modules need to be as independent as possible

Get back to me when you're over a 100 😉 New problems pop up. 🙂

Well I belive you :D new app has modules per feature and per layer so 100 won't be hard to get. I will try gather some numbers then and post it to you ;)

Have you tried Bazel? The setup is a pain, the IDE plugin has issues but that initial one time setup has massive playoffs down the line it's so much more pleasant to use and faster.

I have not, though a few colleagues have looked into it. Massive amount of work...

I got the same problem once. never overused modules anymore.

I know I'm probably preaching to the choir here, but have you made sure all modules' dependencies are handling incremental annotation processing? I'm working on a multi-module proj by myself and once I fixed this, build times *significantly* dropped.

I agree man

Make sure your Gradle jvm Max heap is not too high. Once it is too big everything slows down, something to do with CPU cache miss etc... Also what CPU are you running this on, is it cooled properly? Is the project structured so it can actually build in parallel and from-cache?

Seriously. A 4gb heap vs a 2gb heap can double the build time due to the overhead. Give the timeline a look in a scan, is it actually able to parallelize or is the project dependency tree forcing everything to run in serial? On a laptop? It's probably at thermal limits and slow?