Skip to content

Add GitHub importer - #1658

Merged
jecisc merged 11 commits into
developmentfrom
github-importer
Sep 10, 2026
Merged

jecisc merged 11 commits into
developmentfrom
github-importer

Conversation

@jecisc

@jecisc jecisc commented Sep 5, 2026 •

Copy link
Copy Markdown
Member

This PR propose to add a github importer for Moose.

This is a first version with limitations.

  • We can search projects on github based on their names and owner. Later we could add more filters
  • For now, it works only for Java, later I'd like to add support for other languages
  • For now it only uses the latest version of VVJ and does not let the user provide additional import info such as dependencies path or jdk version

In the future I'd like to add pragmas to each Famix importer to make them discoverable and we could delegate the request of additional info to the importers depending on the language.

Also we can provide a github token to lift the API limit rate

But it will be for another time.

image

@jecisc

jecisc commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Also, for now I used FamixJavaFoldersImporter. But I think that @Gabriel-Darbord is working on import also with Nexus? Maybe in the future we could unify

@Gabriel-Darbord Gabriel-Darbord left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jecisc, this doesn't conflict with MooseNexus, they would even go well together.
For this review, I didn't go into technical details or tried it myself, it looks sound and you've probably tried it already, issues will arise if need be :)

I just have two remarks regarding where data is stored locally, and the choice of archive format.

Also, it's out of scope of this PR, but instead or in addition to having to use a token and potentially overwrite user credentials, GitHub has its own gh cli which could be used to make queries.
We would need a better infrastructure around it though, rather than just using LibC, so that's future work.

Comment thread src/MooseIDE-Github-Importer-Tests/MiGithubImporterTest.class.st Outdated
Comment on lines +76 to +81
archiveReference := self cacheDirectory / (aVersion folderName , '.tar.gz').
archiveReference parent ensureCreateDirectory.
(ZnClient new
url: (repository downloadUrlForVersion: aVersion);
downloadTo: archiveReference pathString) ifFalse: [ self error: 'Cannot download the sources of ' , aVersion displayName ].
LibC runCommand: 'tar -xzvf "' , archiveReference pathString , '" -C "' , aFolder pathString , '" --strip-components=1'.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that we could also use the .zip archive, which is more universally supported (I'm doubtful about Windows support for tarballs).
There's even a Pharo-native ZipArchive class that can be used instead of LibC.
At least, the return code of the command should be checked to give a clear error in case of failure.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion on this point.

Maybe the strongest way could be to use a zip, use libC to unzip since it's way faster than Pharo, and fallback to ZipArchive if there has been an error? (like an OS without unzip installed)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked and tar seems to be shipped in windows since Win10 1803 (bsdtar). This could be enough?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I guess tar is fine.
I would still verify the result of the command to give a clear error in case anything happens, rather than letting it fail silently, then getting potentially hard-to-debug errors down the line.

@jecisc
jecisc merged commit 37c4778 into development Sep 10, 2026
4 checks passed
@jecisc
jecisc deleted the github-importer branch September 10, 2026 14:27
@Gabriel-Darbord

Copy link
Copy Markdown
Member

I notice you used github-moose-cache/MooseIDE instead of MooseIDE/github-moose-cache I proposed, though I also said that doesn't really matter.
My logic is that MooseIDE is the only project that uses github-moose-cache so the cache belongs to MooseIDE, instead of the cache being shared across projects including MooseIDE (unless there really are other projects that use it).

@jecisc

jecisc commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

For now I did not add a MooseIDE level folder but we could indeed

image

@Gabriel-Darbord

Copy link
Copy Markdown
Member

Ah OK, I was misled by a test that made me think this was part of the structure.
It's fine though :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants