jreinhardt/handkerchief

so that pip install does all the magic

:+1: for this awesome project there's a little tweak I would suggest: make the default view a full-width view of the list of titles of issues only (all issues themselves visible). then if you click into an issue the issue-list can return to the (current) small size, and show that issue. then you should be able to "close" the open issue to view the wide issue list again.

I am glad you like it! For me this behaviour is not important at all. I will probably not implement it. But if someone does, I am happy to accept patches.

minor bug 

I can not reproduce this behaviour here (Firefox), but I added a title tag with useful content to the default theme, so your browser should not need to guess a title anymore.

For my purpose, having access to closed issue is useless. And then it is useless to download hundred issues where only few issues are opened. In fact, it may be the default behavior..

I added a commandline switch, but left the defaults as they are.

Thank you :-)

Thanks for this @jreinhardt !

@jreinhardt when running handkerchief --state open woothemes/sensei It simply return this: ``` usage: Download GitHub Issues into self-contained HTML file [-h] [-o OUTNAME] [-l LAYOUT] [-q VERBOSE] [--state {all,open,closed}] [--local] [-a] [--no-local-avatars] [reponame [reponame ...]] ``` Is there something wrong with my command ?

Hmm, that is strange, works here as expected. Can you give some details about your OS, python version? Does it work without the state switch?

@jreinhardt I'm running OSX Yosimite. After trying it on an ubuntu server this worked nicely: python handkerchief.py woothemes/sensei --state open

So this works on OSX as well or on ubuntu only?

I can't confirm OSX yet as I'm getting this : ``` $ handkerchief woothemes/sensei --state open There is a problem with the request https://api.github.com/repos/jreinhardt/handkerchief/contents/layouts/default/default.json <Response [403]> ``` I pressed Ctrl-C ( CMD-C ) to quit the script while it was in operation and got the above when retrying. I need to restart the system and test it again. **On ubutnu:** handkerchief --state open woothemes/sensei Returns this: ```` usage: Download GitHub Issues into self-contained HTML file [-h] [-o OUTNAME] [-l LAYOUT] [-q VERBOSE] [--state {all,open,closed}] [--local] [-a] [--no-local-avatars] [reponame [reponame ...]] ```` But this works on ubuntu : python handkerchief.py woothemes/sensei --state open

For unauthenticated API calls GitHub poses a rate limit of 50 or 60 requests per hour. This usually is not a problem, but when testing or debugging handkerchief, you hit this pretty quickly. In this case on gets an error message just like you got. So it might be that it just works again after a while. Alternatively you can use the -a switch to authenticate your api calls. Edit: The script seems to be confused by the script name. I am not sure why and how this happens, I will see if I can find something.

I just realized that you probably use the bash function that is described in the README. I never tested this (someone else suggested it for use on MacOS), and it indeed seems buggy. Can you try to remove the whole function from your .bashrc file and replace it with alias fftr='python path/to/handkerchief.py' If this works for you on MacOS, I will update the readme accordingly.

If some comments contains html code, it is interpreted directly by the browser. And if it contains a script element with JS code, the JS code is executed! It mustn't be done.

It fix a design issue when code in comments contains long lines

Now the inline code blocks have a style and there is a different one for the blocks inside the pre.

It uses normal checkboxes like github now.

As layed out in #6

did that. If called without explicitly given repos, handkerchief now not only scans for repos in the current directory, but also checks for html files with a repo marker and updates from the corresponding repos.

Hey @jreinhardt, awesome, but is it possible, that it produces an error when trying to init one in an empty directory? I get this error: ``` Traceback (most recent call last): File "/Users/lukasoppermann/projects/knowledge-base/handkerchief/handkerchief.py", line 189, in <module> reponame = branches.values()[0] IndexError: list index out of range ```

Done.

Perfect, now it works just fine.

Fixes #13

Is there a way we can add assigned to xyz as filters?

Yes, I made the necessary changes in code and template * [ ] add nonugly styling to assignee filter

Cool, I will take care of the styling.

@jreinhardt I might have not seen all layout bugs so please tell me if you find more.

I noticed that somethings are not parsed according to the github flavour of markdown. - [x] `- [ ]` for e.g. should be a list-item with a checkbox - [x] Also maybe links could be automatically turned into real links. - [x] code blocks could check for language and add it to the code tag as a class so we can add a syntax-higlighter later on. - [x] add a class depending on code type - \`code\` is inline - \`\`\`optionally-language <linebreak> code \`\`\` is block (which can have a language) - [ ] is it possible to get the link to issues from github and add that to the comment as well? (Could link to the issues offline, like this: #11 )

Yes, the Javascript Markdown renderer that I use does not speak GFM. What do you think, should I keep Markdown conversion in Javascript ([marked](https://github.com/chjj/marked) seems to be the library to use), or should I move it to python, at the cost of additional dependencies (have to do more research about what to use), or should I let GitHub render it using its [Markdown API](https://developer.github.com/v3/markdown/) (at the cost of running into rate limit issues much earlier, as each comment is one api call). The third possibility is probably the most accurate one, as it will also take care of issue links and stuff. But I like the fact that at the moment we can get away without authentification for most repos.

I would try marked first, I guess it is the easiest way to go. It would be awesome to keep github api out of it. If #links don't work I guess one can live with it, or write some additional logic in js afterwards.

Marked does a bit better than markdown.js, so Links are now taken care of. It also seems to be a bit more customisable, so I can try to bend it to our will. The first post of this issue broke the layout as mentioned in #12 so I edited the offending part.

Perfect. Links wor like a charm. As for the HTML couldn't you just run a find&replace and replace all < with &lt;? This should stop all html from working. As for the todo-lists I found this in a post of the repo But I can not get it to work. ```javascript renderer.listitem = function(text) { if (/^\s*\[[x ]\]\s*/.test(text)) { text = text .replace(/^\s*\[ \]\s*/, '<i class="empty checkbox icon"></i> ') .replace(/^\s*\[x\]\s*/, '<i class="checked checkbox icon"></i> '); return '<li style="list-style: none">' + text + '</li>'; } else { return '<li>' + text + '</li>'; } }; ``` Maybe with the renderer we can fix most of the other points as well?

The code blocks with language class seem to work out of the box. I did not need to do anything. Also I got checkboxes to work. The code above works, but only for tight lists, e.g. those without empty lines between the items. For loose lists like the one in the first comment to this issue additional code was necessary. There is no styling applied to the checkboxes yet.

Fixed styling in #16 There is one problem though: lists should be separate if divided by a line break. Right now the below 2 lists would result in only one ordered list. if I have an unordered list * list * list 1. test 2. test

This is a problem in marked and should be reported there.

I reported it here https://github.com/chjj/marked/issues/340 hope they fix it, any easy workarounds otherwise?

None that I am aware of.

Okay, I thought maybe the main list fn could be replaced as well just like with the checkboxes. Do you think we should use highlight.js like in the example? Could be nice for big chunks of code. https://github.com/chjj/marked#highlight

Ok. You just need to add highlight.js to the default layout folder, add it to default.js and modify the js code.

Perfect, I will look into it.

Hey @jreinhardt I tried it here https://github.com/lukasoppermann/handkerchief/commit/2db62005739e22445721726c9d77b568b19771bb but I did not get it to really work. Could you check it out? Also something likes this is not shown. Can you change `<` to `&lt;` so that code is not interpreted? ``` <? echo "test"; ?> ```

The problem was that the version of highlight.js that you used did not contain any language definitions. I replaced it by a packed version downloaded from [the download page](http://highlightjs.org/download/). The second issue seems to be another problem with marked.

Perfect, it works like a charm, thank you.

Small thing, but it would be nice to show something like "downloading issues ..." while the python script is running. And maybe it could open the file in the browser once it is done downloading?

I don't like it when scripts are talky, but I guess for use in other scripts or Makefiles I can add a "quiet" parameter to suppress the output. But I really don't like it when scripts do stuff that potentially steals window focus. So I won't do the opening thing. Plus it is a trivial ```handkerchief.py && browser issues-reponame.html```

Okay, I do understand this I just thought the script broke in the beginning when nothing happend for a while. This is why I think a "downloading issues ..." or something would be helpful. Opening the file in the browser can easily be done by adding it to the alias function in the profile anyway.

Hey, I am working on fixing the styles. I noticed the following that should be changed: - [x] a user avatar class thingy should be create for the issue (and the issue content) as well. (Its done for comments only, currently) - [x] the comment bubble in an issue (with issue content) should only be displayed if there is a comment (I did not get this to work, it should be something like `{% if issue['body'] %}` )

Do you have an example for such an empty comment?

Sure, its something like this one https://github.com/lukasoppermann/html5nestable/issues/2

The Jinja expression is exactly what you thought it should be.

Sorry to annoy you with all the issues. :) It would be great if the script would work a little bit more with the information it gets from git. - [x] name the file "reponame"-issues.html (e.g. hankerchief-issues.html) this would make it possible to have multiple issue files of more than one project in one directory and maybe even later on make it possible add a repo navigation to the pages (probably using javascript) - [x] automatically detect if called from within a repository and get the repository name so this function is obsolete https://github.com/jreinhardt/handkerchief/pull/7 and one could by default just call handkerchief.py without any arguments if within a repo.

This request changes the following things: - template has been renamed to `default.html` - variables have a new syntax `{variableName}` - added added new styling and moved everything to `style.css` - labels need to get the color from the api call assigned to the `{color}` in the style attribute on the bg tag I think we should go for a different logic: 1. get data from api 2. render all elements with data (maybe use data-labels, data-milestone and data-id) 2.2 only the elements with the class visible are actually visible 3. Instead of the js building elements on the fly, it just finds the correct elements (using the data-) and adds the visible class to those. We should stick to your no-framework idea, since we just need to use `getElementByClassName` to get the and can add/remove the classes using the `classlist` api. (Can you move all the js to a separate file, which would make changing and debugging a lot easier). - Labels and milestones should be deselected when clicking on a selected item. Closes #6 #2 @jreinhardt you think it you can work with this?

Could you maybe: 1. Explain how the templating works? (Or link to it if it is an external tool) And maybe add the following to the issues (inside the comments div): - [ ] Issue title - [ ] State - [ ] Author - [ ] Issue number - [ ] Issue Link (helpful when not offline) Also maybe you could replace the main Title (h1) with the name of the repository (maybe even with the link to the repo on github). Thanks.

Ok, maybe we should step back a bit and think about the general approach to this problem first. I am currently not so sure if the current way how handkerchief works is really a good idea. Currently what handkerchief.py does is that it polls the GitHub API for all issues and comments on issues. It then dumps the resulting JSON into the template, where there is a bit of Javascript hackery (see line 49-151) that extracts the infos from JSON data and displays it. So almost everything is done in Javascript. An alternative would be to do much more preprocessing in python. I did not use jQuery or any other JS framework apart from Markdown rendering, because I somehow had the fixed idea to put everything in the python file and wanted to keep dependencies minimal. Now I realized that there is no need to do that, instead one can poll the GitHub API for such content, as handkerchief is only useful when GitHub can be reached anyway. This way I can avoid a lot of the clutter that I have in handkerchief.py right now. I will probably rewrite everything from scratch (which is not a problem, as it is only a few dozen lines of actual code). So maybe the best approach to continue would be if you could mock up a HTML file that looks like you want to have it, and use whatever JS framework or other magic you need for making it awesome. With that I can rewrite handkerchief to obtain and plug in the data in the needed form.

Hey this sounds like a good plan, maybe we should sketch out some ideas for where it could go and decide what we want to do and what we need. ## Usecases I can think of two usecases: 1. A dev wants to pull all issues for a certain repo (open and closed) to his machine to access it offline. 2. A dev wants to have 1 directory, (for e.g. issues) which holds multiple issue-micropages from different projects (project1-issues.html, project2-issues.html). ## Keeping it simple We should split our approach here. I would say the way of installation should be: 1. get dependencies (python, ...) 2. clone repo to somewhere on your machine 3. add a shortcut to call the script to the bash_profile, etc. This means we can potentially put as many files as we want into the repo (so we can split template, javascript and css into separate files) **BUT** the output should stay very simple because those files will be in the users folder structure. Here I like your approach of one file storage. So one file per repository is what I would think. ## Vision What I would like this to turn into is the following: 1. The scripts pulls issues offline into a single html file (usecase 1) 2. A js script in the html files looks for other files in the folder and links all files together so a user can browse multiple issue files within one folder (usecase 2) ## Convenience Maybe you could do this: `handkerchief.py username/repo` pulls the repo down the the current folder and adds a comment to the very top `repository: username/repo `; `handkerchief.py update` checks for html-files in the current repo and reads the first lines which would be a comment stating the string `repository: username/repo ` if this is found it performs a `handkerchief.py username/repo` request (replacing this file with the new version). This would mean once you pulled a repo down you would just need to do a handkerchief.py update and everything would be up to date. I know this sounds like a lot of work, but maybe I think the changes are actually not sooo big :) and I would try to do the js work that is performed within the page. What do you think?

## Usecases I fall under use case 1, but I believe supporting use case 2 is not too much work, so that is fine with me. ## Keeping it simple Installation like you sketched is fine with me. For people that don't want to fiddle with their system paths it can probably even be shortened to 1. Get dependencies (python, requests) 2. Drop handkerchief.py into a repo that you want to get offline issues for ## Vision Is there any problem with realising use case 2 by allowing to specify multiple repos on the commandline? In this way one would avoid the parsing effort. Plus I believe a js script in a html file can access the filesystem from the browser. ## Convenience I would put this a bit lower on the priorities list. The mechanism you outline seems viable, but I would prefer to start with the minimal one-repo functionality.

Yes, the "Convenience" part was though of as a lower priority anyway. The *problem* with specifying the the repos in the commandline would be, that whenever I want to update it, I would need to specify all of the repos again. (If I understand you correctly). This would not be a problem if we would have an update function. But it would still pose a problem if I want to easily add another repos issues, correct? (I might be wrong though, which would be handy). *Plus I believe a js script in a html file can access the filesystem from the browser.* Do you mean you do NOT believe it possible? I think it should work with the html5 file api. But of course if it can work within the parsing step this would be so much more handy.

The problem might be not so critical if one uses a build system like make anyway, handkerchief with all the arguments could be just another target. But as I said, we can look into that once we have the basic functionality done, it should not be too much work. Yes, sorry, I forgot a no there. And you are right, the file api seems to allow that.

Okay, I don't really get what you mean with the build system, but if you think it can work conveniently, lets get stuff moving. I will start the template (I will probably be offline until sunday, but I should have a little something by then).

With build system I mean something like make or grunt.

Ahh, yes, that is true, but it does create a need for facilitating something like this, which in turn means more setup work. But we will see.

When I run ``` python ~/projects/knowledge-base/handkerchief/handkerchief.py lukasoppermann/project ``` **I get this error:** ``` Traceback (most recent call last): File "/Users/lukasoppermann/projects/knowledge-base/handkerchief/handkerchief.py", line 237, in <module> reponame = branches.values()[0] IndexError: list index out of range ``` I tried removing this part like this ``` #try to figure out repo from git repo in current directory reponame = None try: remote_data = subprocess.check_output(["git","remote","-v","show"]) # branches = {} # for line in remote_data.split("\n"): # if line.strip() == "": # continue # remote_match = re.match("([a-zA-Z0-9_]*)\s*git@github.com:([a-zA-Z0-9_/]*)\.git\s*\(([a-z]*)\)",line) # if not remote_match is None: # branches[remote_match.group(1)] = remote_match.group(2) # # reponame = branches.values()[0] # if "origin" in branches: # reponame = branches["origin"] except OSError: pass ``` Which results in this error ``` <https://api.github.com/repositories/3535672/issues?state=open&filter=all&direction=asc&page=2>; rel="next", <https://api.github.com/repositories/3535672/issues?state=open&filter=all&direction=asc&page=2>; rel="last" Traceback (most recent call last): File "/Users/lukasoppermann/projects/knowledge-base/handkerchief/handkerchief.py", line 253, in <module> '<https://api.github.com/repositories/([0-9]*)/issues\?state=%s&filter=all&direction=asc&page=([0-9]*)>; rel="last"' % state) File "/Users/lukasoppermann/projects/knowledge-base/handkerchief/handkerchief.py", line 209, in get_all_pages print request.headers["link"] File "/usr/local/lib/python2.7/site-packages/requests/structures.py", line 77, in __getitem__ return self._store[key.lower()][1] KeyError: 'link' ```

Hi, apparently github changed the behaviour in case there is only one page of issues. fcd7e44 fixes the second error. The first error seems to be related to the way I try to figure out which branch to consider. Could you paste the output of git remote -v show in the repository? And the output of git --version Greetings

Hey, sure thing, I am running ``` git version 1.9.3 ``` and the output for -v show is ``` origin https://github.com/lukasoppermann/mark.git (fetch) origin https://github.com/lukasoppermann/mark.git (push) ```

Ok, the first error is caused by the fact that I only use git@github type urls, so the https urls were not recognized. I hope 2a57701 fixes that. Could you try that?

Yes, that fixes it, thanks a lot.

This is a test comment

And another one

Now there are enough real comments

Are you planning on adding any styling to it? Otherwise if I find the time to add some styling, would you be interested in a pull request? What would be neat, is to add some `active` classes to the active menu items so that special styling can be applied to those for better orientation. Did you think about adding templates and external css files (which would be merged into a single file when retrieving the issues) to make it easier to develop layouts for the script? This should not provide any issues, since you could merge all files in the issues.html, but keep separate files in the script repository.

I would like to add styling, but I suck at web design. And from your GitHub Profile I see that you are very good at this, so yes, I would be very much interested in a pull request! Adding classes to certain items should be easy, so if you are missing something, just tell me, I will try to implement it. I had planned to keep everything simplistic in one script file to allow to just pop handkerchief.py it into the directory of any project and use it without fiddling with paths or worry about system wide installation or anything. But this has the downside that one has to include all js and css in the python file which is a bit of a mess.

Hey, cool, I will look into it. I would definitely like it if all sidebar items that are "active" have a css class like for e.g. `active` so that they could be style differently. That would be something I am missing. I do understand the idea of keeping it simple, but I would guess people clone the repo anyway and just put this one folder somewhere, right? So it would not be such a huge problem. The benefit would be, that people like me, who do not no anything at all about python, could fiddle with the html, css and js to edit the output. So maybe you can consider changing this?

Ok, I added code that adds the active class for the active filter. Actually I dont know how people use it. Actually I am not sure if anybody apart from me and you actually uses it. What I could do is provide commandline switches for providing a template, and use the included one if no template is given. This would make development of layouts easy and still preserve the one-file idea.

I had this idea with adding the switch as well. Sounds like a very good plan. :+1: I do know that there are not a lot of good tools to store github issues offline (actually I did not find anything apart form your script that was working).

Ok, I added the switch, and commited a example template with the standard style which you can use as a starting point for improvements.

Perfect, I will add some more issues for other features I think could be helpful, an start working on this probably this weekend. Actually I do misunderstood you. Couldn't you check if there is a default.template file and only if none is present use the internal one? So by default it would use the file? Also the path should be "same as handkerchief.py -> search for default.template" because I have it installed "globally" but I want to download the issues per repository. This way I can keep doing it an the default template will always be used.

Closed, see #7