My journey with HTPY

[ggranger]

Hi everyone!
I just spent the last month migrating an entire project to HTPY and I wanted to share this journey with you, probably it can be useful to others but more than that, it's a way for me to show my appreciation to the creators and the maintainers of this wonderful project.

The project and the motivation

My project is a side-project that I've been maintaining alone for years. Stack is:

• Framework: Flask + Jinja2 templates
• Internationalization: Babel through Flask-Babel extension
• Forms: WTForms through Flask-WTF extension
• Frontend library: Fomantic UI

So why move to HTPY in the first place ? It all began with the announcement of the new FastHtml framework and one of the pillars for this framework is HTML generated by Python code through their own FastTags system (HTPY equivalent). My first reaction was WTF, why would you do that instead of using good old templates? Then I wondered if I was truly happy with my templates and soon realized that in fact they were pretty limiting:

• Not being able to do actual python code in templates is enforcing the separation of concerns which is good, but to circumvent that limitation you are often forced to pass context to templates that would be already available otherwise. A good example of that is accessing some Enum values.
• Component management is done through macros that look more of less like python but less readable and without the structure allowed by native python. I'm looking for going all in on server side components with the help of HTMX so making things even worse.
• And finally, there's the type checking and linting that is completely missing. Parameter expected by a given template are impossible to tell, even more when you are doing inheritance ("extends") or inclusions ("includes"). So you often realize at runtime that you're missing a parameter, or sometimes that a parameter is for some reason never used.

I decided to go with HTPY in particular for multiple reasons:

• The website is amazing, you feels that a lot of care has been put to describe and document the project
• The syntax is neat (though I think it can be improved, more on that later), with IDE completion, linting, code formatting, etc...
• The project is very much alive with regular contributions
• The maintainers are friendly, reactive, and very open to community proposals

The migration process

So I decided to migrate my ~60 templates to HTPY. 2 weeks "full time" work during my summer holidays.
I won't lie, it's been painful, even though using the html2htpy utility made things a lot easier. Despite that, the process was still cumbersome and involved a lot of manual and repetitive tasks:

• run the html2htpy for a given template
• copy/paste in my favorite IDE
• go through all the Jinja specific instructions not managed by the tool to properly re-implement them
  • conditional {% if ... %}
  • loops {% for ... %}
  • Babel strings {{ _("...") }}
  • etc...
• take the opportunity to clean the templates:
  • over the years, I ended up with some bits of CSS and JS that had nothing to do in my templates in the first place
  • doing actual Python actually allowed me to remove some cumbersome designs that were initially implemented

All in all, migrating a template would take me between 30 minutes and 3 hours depending on the complexity, but I can tell you it was well worth it.

The component management

The HTPY website recommends managing components with functions. This was my initial approach and even though it was okay in the beginning, I soon realized it would be limiting to re-implement the same kind of inheritance behaviour I was liking in Jinja2.
So I ended up taking a class based approach. Starting with creating a base View class that all my templates would inherit from:

from htpy import Node, render_node

class View:

    def __init__(self, *nodes: Node) -> None:
        self._nodes = nodes

    def __str__(self) -> str:
        return self.__html__()

    def __html__(self) -> str:
        return render_node(self._nodes)

Then I could create a basic template like so where title and content are fragments that can be overriden :

class PageView(View):

    def title(self) -> str:
        return "My title"

    def content(self) -> Node:
        return None

    def __init__(self) -> None:
        super().__init__(
            html()[
                head()[
                    title()[self.title()]
                ],
                body()[self.content()]
            ]
        )

For a specific page, I will then subclass a given template:

class HomePage(PageView):

    @override
    def content(self) -> Node:
        return p()[f"Hello {self.user.name}!"]

    def __init__(self, *, user:User) -> None:
        self.user = user
        super().__init__()

Now in my controller, when I want to render a full page, I will do:

return str(
    HomePage(user=user)
)

Or for a specific component or fragment:

return str(
    HomePage(user=user).content()
)

If for some reason I need to render multiple components that are not part of a single Node, I will do the following (can be useful in some situations with HTMX):

page = MyPage(param=v)
return str(
    View(
        page.component1(),
        page.component2()
    )
)

I find this class system very flexible, readable and easy to maintain, highly recommend HTPY users to take a similar approach instead of the function based approach.

The naming conventions

One annoyance I had while migrating to HTPY is the naming convention for tags. They are in lower case words for standard elements and lower snake_case for custom elements. The things is that it makes it hard to read and most of all creates some naming overlap with some of your variables if you don't pay attention.

For example, all my WTForms instances were assigned to a variable named form. No luck that there is also an HTML element named form. It took me hours to solve this issue. The problem can happen (and I'm sure will happen again) for a lot of elements like a, label that I often use as variable names; the same goes for input that conflicts with the native Python input() function.

My proposal here would be to allow (and not replace because I realize it would be a big breaking change) for having tags in PascalCase.
So div could be aliased to Div, my_element could be aliased to MyElement (and utimately translated to my-element when rendered as HTML).

Multiple reasons for that proposal:

• easier to read (this is subjective)
• closer syntax to JSX components that a lot are familiar with
• no more (or less??) conflict with variable names that are in general in lower case.

The datatypes

In the process of migrating, one error I soon encountered was when trying to display a badge of unread messages. The count was an int which is not a valid datatype a child Node. So you cant do span[count] but span[str(count)]instead. I also found myself trying to render other data types that obviously couldn't be handled by HTPY such as my WTForms fields or my datetime attributes.

My proposal here would be to introduce some kind of custom serializer to avoid having some str()all over the place.
It could work like the native Python json module; the json.dumps() function accepts a cls parameter that you can initialize with a custom subclass of JSONEncoder to allow serialization of complex Python objects to json.

The existing htpy.render_node() function could benefit from having a similar pattern to allow for complex serialization that would otherwise end up being repeated in several places. Just an idea, probably there are other ways to achieve this or is not part of the project's vision.

The error management

This leads me to a final complaint about errors that are very cryptic when htpy cannot properly render your code for whatever reason, making it very hard to troubleshoot what the problem is and where the problem is. So I often ended up commenting code until it works and then reactivating until I find the issue, not very efficient.

Conclusion

Sorry for the long rant, but I would like to express to the community and in particular to the creators and maintainers how valuable the HTPY project has become to me. It made my own project much cleaner and modular than it used to be.
Yes I have a few complaints but nothing is a blocker for me at this point and I'm aware that the project is still very young and has lots of room for improvement still.
So thanks for what you have done with HTPY!

[pelme]

Thanks a lot for writing this down, it is really useful to me and hopefully others too! Will reply in more detail, I think there are good solutions to a lot of the problems you describe or that they can be documented/worked around :)

[pelme]

Hi everyone! I just spent the last month migrating an entire project to HTPY and I wanted to share this journey with you, probably it can be useful to others but more than that, it's a way for me to show my appreciation to the creators and the maintainers of this wonderful project.

Thanks for taking the time to share this. It is very valuable feedback!

The component management

The HTPY website recommends managing components with functions. This was my initial approach and even though it was okay in the beginning, I soon realized it would be limiting to re-implement the same kind of inheritance behaviour I was liking in Jinja2. So I ended up taking a class based approach. Starting with creating a base View class that all my templates would inherit from:
...
I find this class system very flexible, readable and easy to maintain, highly recommend HTPY users to take a similar approach instead of the function based approach.

Thanks for sharing this. Django has class based views, I personally use function based views in 95% of the cases. To me, following the inheritance flow of CBV and multiple levels of inheritance is very hard. I know a lot of people like class based views. Personally, I find combining functions much easier to follow. We have built quite sophisticated UIs in our code base with just functions.

That said, one of the primary reason to use htpy is to leverage the full power of Python. Using functions, classes, generators, modules, the dynamic nature of Python is totally fine and is what makes htpy so powerful. Just like the same arguments of functions vs classes/inheritance can be made for any code, the same applies here. Different projects/backgrounds/ideas are certainly possible and there are no single "right way" to structure this code.

I would be open to adding a class based approach to components to the docs. Defining __html__() on the class breaks streaming (https://htpy.dev/streaming/) though which would be annoying and we would have to find some way to work with that. Please open a separate issue/discussion if you are interested in pursuing it!

The naming conventions

One annoyance I had while migrating to HTPY is the naming convention for tags. They are in lower case words for standard elements and lower snake_case for custom elements. The things is that it makes it hard to read and most of all creates some naming overlap with some of your variables if you don't pay attention.
...
My proposal here would be to allow (and not replace because I realize it would be a big breaking change) for having tags in PascalCase. So div could be aliased to Div, my_element could be aliased to MyElement (and utimately translated to my-element when rendered as HTML).

I would like to avoid another way of naming/defining the elements to stay consistent and not give multiple ways of doing the same thing. It would be annoying to have a mixture of div and Div. I like it lower case since that is the way html elements are typically written these days. :)

Multiple reasons for that proposal:

* easier to read (this is subjective)

* closer syntax to JSX components that a lot are familiar with

* no more (or less??) conflict with variable names that are in general in lower case.

I have started using import htpy as h in a lot of places. It avoids name clashes and also separates "basic elements" from components when you read the code. What do you think about that?

Btw, React also does lowercase for the HTML elements. JSX uses lower or upper case of the first character of a JSX element to determine whether it is a HTML element or component. When React moved on from class components to functions/hooks, you still had to define your function components with an upper case letter. function Hello() { return <h1>Hi!</h1> }. To me, it is just a limitation/how JSX works. It is very rare to see JavaScript functions named using PascalCase in other places. So I don't think React is a good role model in this case!

The datatypes

In the process of migrating, one error I soon encountered was when trying to display a badge of unread messages. The count was an int which is not a valid datatype a child Node. So you cant do span[count] but span[str(count)]instead. I also found myself trying to render other data types that obviously couldn't be handled by HTPY such as my WTForms fields or my datetime attributes.

My proposal here would be to introduce some kind of custom serializer to avoid having some str()all over the place. It could work like the native Python json module; the json.dumps() function accepts a cls parameter that you can initialize with a custom subclass of JSONEncoder to allow serialization of complex Python objects to json.

The existing htpy.render_node() function could benefit from having a similar pattern to allow for complex serialization that would otherwise end up being repeated in several places. Just an idea, probably there are other ways to achieve this or is not part of the project's vision.

• Not being able to pass int's was really annoying, that is added now: Support int as attributes and children #52
• Forcing str() on all objects like Django templates does would indeed be convenient. I did not add that though since accepting any object where str(x) would be allowed would force the type of Node to be Any. It would make typing much worse since you could just accidentally pass any object that would then just end up showing its repr. While the extra str() calls annoy me too, I think they are worth it over the alternative.
• Can you open a separate discussion/issue about WTForms with more details? I render Django forms with htpy and it works since Django forms uses the __html__() protocol. I would think that WTForms does the same?
• I think having a formatter/serializer could be tricky and may hide/make it a bit unclear how a particular object would be formatted by just looking at the code. Would love to discuss more though so please also open another issue/discussion if you have more ideas on it. I do think just using regular functions and f strings go a long way here! `f

The error management

This leads me to a final complaint about errors that are very cryptic when htpy cannot properly render your code for whatever reason, making it very hard to troubleshoot what the problem is and where the problem is. So I often ended up commenting code until it works and then reactivating until I find the issue, not very efficient.

I agree, I had problems with this too. I will open issues/fix this. Currently the validation of the attributes and children happens at rendering time, not when the elements are constructed. Not all cases can be caught if using streaming with callables or generators but most common cases could raise an error directly and give a normal/nice stack trace.

Conclusion

Sorry for the long rant, but I would like to express to the community and in particular to the creators and maintainers how valuable the HTPY project has become to me. It made my own project much cleaner and modular than it used to be. Yes I have a few complaints but nothing is a blocker for me at this point and I'm aware that the project is still very young and has lots of room for improvement still. So thanks for what you have done with HTPY!

Again, thanks for taking the time to write this, I am glad htpy is useful to you too! :)

[ggranger]

Thanks @pelme for your detailed answer.

Actually the most important topic to me remains the error management. I don't use streaming so I can't really tell but from my experience with htpy, this has been the most painful part (even though luckily I didn't have much to deal with). Happy that you are looking into it.

Regarding the naming convention of tags, I have tried the import htpy as h pattern but I couldn't live with it, made my code much less readable. I understand that adding yet another naming convention based on a different case is not an option for you even though it would offer more options without breaking too much the current logic IMHO.

Probably I'm going to try and change my own imports such as from htpy import div as Div and see how I can live with it (and how it improves my variable name conflicts).

Thanks again for your reactivity, love htpy!

[pelme]

You can also do your wrapper module with something like:

htpyupper.py

import htpy
def __getattr__(name: str) -> Element:
    return getattr(htpy, name.lower())

And then from htpyupper import Div if you like. just a thought, did not test it. 😆

[ggranger]

Hey @pelme,

I spent quite some time playing with your proposal and I'm very happy with the result.
I will leave here the result from my work in case it's interesting for other people and I would definitely appreciate that you consider at some point adding this as part of htpy given how much flexibility it's adding without any increased complexity (htpy should internally only handle snake_case elements as it's doing now but only open the external API to accept UpperCase elements imports).

So my htpy wrapper looks like this tags.py

import re
import htpy

def __getattr__(name: str) -> htpy.Element:
    words = filter(None, re.sub(r"([A-Z])", r"_\1", name).split("_"))
    snake_case = "_".join(words).lower()
    return getattr(htpy, snake_case)

Yup, fairly simple. And it does everything that I need from it, here's my associated unit tests for reference:

import htpy as h
import app.views.tags as t

def test_div() -> None:
    assert str(h.div) == str(t.div) == str(t.Div)

def test_img() -> None:
    assert str(h.img) == str(t.img) == str(t.Img)

def test_p() -> None:
    assert str(h.p) == str(t.p) == str(t.P)

def test_my_component() -> None:
    assert (
        str(h.my_component)
        == str(t.my_component)
        == str(t.MyComponent)
        == str(t.My_Component)
    )

def test_a_component() -> None:
    assert (
        str(h.a_component)
        == str(t.a_component)
        == str(t.AComponent)
        == str(t.A_Component)
    )

[pelme]

That is cool, I am glad you found it useful! Thanks for posting the complete solution.

I still am hesitant to add it to htpy. For better or worse, the tags are lower case now. In this case, I think it is more important to have one single way of doing it even though I can see the arguments for both div and Div.

I am very happy that you and anyone else can drop in a tags.py file with a few lines of code and get it the way you like in you project!