Migration from custom container library to augmented EASTL

Okay, I’m gonna extract this question into separate thread to keep discussion on-topic.

Currently there’s very good opportunity to move Urho from custom containers to proper 3rd party container library. Most of the porting is already done.

The issue is that this change is breaking. There are adapters that covers common cases but they don’t provide 100% backward compatibility. And names are not in CamelCase /outcry of shock/.

What this migration offers:

• Container library maintained by companies instead of custom-made library that nobody maintains;
• Move-semantics friendly containers;
• Containers that don’t allocate memory dynamically;
• Algorithms;

Why one may want this change?

When a person is used to work with std container library in its entirety (more than just vector.push_back), using custom Urho Containers becomes a pain. There’re so many things missing. I have refactored Urho3D::Vector<T> two times when I needed something from it and it’s not even close to “enough”.

Small buffer optimization? Nope. Allocators? Nope. Algorithms? Nope. Containers except vector and hash map? Nope. Move semantics? Barely. Spans and string views? Nope.

Urho is outsourcing physics, navigation, image decoding and tons of other tasks. Why it cannot outsource container library too?

So, what do you think? Is it worth it?

Are there any tests comparing Urho and EASTL containers? As far as I understand, Urho3D containers are designed to binding with scripting languages and work fast. Who forbids users to use their own containers in their projects? I am not against progress, however, the loss of binding to scripting languages is a regression. C # cannot replace scripting languages because you cannot quickly call managed code from native code. You can only quickly call native code from managed code.

When I checked last time, script bindings are not heavily dependent on exact container types. They will need some revise, but I think it’s doable.

About performance… measuring performance properly is hard task that requires time and effort. I may try it when I have time for it.

I’m totally voting up for EASTL, I love how Urho3D containers work right now, but the couple of times I had to extend with functionality that’s already in STL I felt like I was failing a job interview. I don’t mind spending 2 days refactoring my code replacing .Push with .push_back.

If this change is decided upon, it would probably be best to make it part of Urho 2.0.

Does this mean you are suggesting to wrap EASTL in Urho classes too? Could there be some backwards compatibility this way or engine modifications averted?

Yeah, great idea.
Aprove this here to pave the way for getting *** in regardless.

Don’t mix topics.
This topic is about this specific feature that I personally missed in Urho for years.

Here is specific scenario:
People want to use Urho. Not some fork, but the master branch. I’m one of these people, but there are more. This is given.
People want to have decent container library in Urho. I’m one of these people, but there are more. This is given too.

What solution do you offer? I expect constructive answer here.

Could you guys pro stl changes divide the issue a little bit? Pls explain the eastl introduction to Urho3D a little technical detailed?

As a proposal, shall we list merits and demerits? Challenges and opportunities? Future plans?

There will be a lot of code changes to Urho as well as your own code. If we gain more, only cost the unification and aesthetics of the engine, its opportunity.

I like the current containers, since they are just simple and good for my usage. But obviously, many people ask more and have their own needs and pains.

Edit: I have to point out that, ***’s Editor is not because of the container changes, it is mainly because of the auther’s hard working. I really respect that. So, I don’t want to mix the ***’s features with the introduction of stl.

There are many people who are silent in this community, including the leaders and ex-leaders, they may have their own private fork or workflow, or have quit using Urho, but for the containers change, please stand out and speak out.

Writing huge posts is always taxing. I may do it when I get energy for it.
// TODO:

I support this idea unless it turns Urho into a gargantuan giant monster where opening the project in the IDE takes 10 minutes , the building process takes 1 hour and the resulted binary is over 30 mb … Also i expect smooth out of box Cmake scripts without any hassle over fixing broken cumbersome scripts. Right now the cmake scripts (at least on windows) are perfect !! I download Urho3d , build it and start working on my game.

Sorry I missed your reply here.

It would be a waste of performance and manpower to do it.

I understand beauty of having all code in single CamelCase style and whatever.
However, one-line trivial wrappers like we have for Sqrt is weird thing to do.

There are CamelCase adapters in augmented EASTL containers to simplty porting. They are disabled by default and do not cover everything.

I too am quite fond of this consistency, and if it would break over this, I think it would be better to use std (to avoid the road to Boost) and change the Urho code style accordingly or leave things the way they are.

If the community is convinced, as long as EASTL itself alone does not break things too much, we may provide some scripts/way to help people migrate their own projects.

As for AngelScript, can we make things automatic? like SWIG for C#? So we just keep lua, AngelScript and scripts based Editor, as long as we don’t have a better alternative yet so discarding any of the parts are not considerable. If people love C#, use it.

So, everyone is happy and Urho is stronger. Am I asking too much (based on the community size and contributor number)?

So you are considering only there two options: break all API compatiblity or do nothing.
This is quite unusual developement position.

I made quick benchmark, feel free to criticize because I’m totally not an expert in benchmarking.
https://gist.github.com/eugeneko/2cbad3b16ca2f8e250d57a17c2d4d207

Run on (12 X 2208 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 Unified 256 KiB (x6)
  L3 Unified 9216 KiB (x1)
----------------------------------------------------------------------
Benchmark                            Time             CPU   Iterations
----------------------------------------------------------------------
Trivial_StringVector_EA        1190020 ns      1199777 ns          560
Trivial_StringVector_Urho      5742792 ns      5719866 ns          112
Trivial_StringHashMap_EA       1692161 ns      1689189 ns          407
Trivial_StringHashMap_Urho     6291083 ns      6277902 ns          112
Real_StringVector_EA           1441662 ns      1443273 ns          498
Real_StringVector_Urho         6115403 ns      5998884 ns          112
Real_SmallStringVector_EA       355737 ns       352926 ns         1948
Real_SmallStringVector_Urho    2116384 ns      2128623 ns          345
Real_HugeHashMap_EA          385838650 ns    390625000 ns            2
Real_HugeHashMap_Urho       1074992600 ns   1078125000 ns            1

You are suggesting to break both compatibility and consistency.

One concern I have with the benchmark is that it may be more of a test of the incremental creation of the strings than of the actual containers. What is you used the same object type for all of the containers (maybe key type as well for the maps, but I understand that might be an issue since the hash function needs to exist).

Also, the suggestion is to break comparability and internal Urho consistency in exchange for a (potentially) better preforming set of containers (and likely greater comparability with external library’s that likely use the stl containers (which I believe share the same function names as EASTL, though not the same classes)).

Proposed compatibility changes differ in scope by the orders of magnitude. Smaller breaking change is always better than bigger breaking change, unless you have strong arguments for doing bigger change.

Consistently is weak argument. Urho way to wrap everything is perfectionism borderline to mental disorder. We have garbage code that is suboptimal and need to be maintained just to have fancy names.

Cmon, why on earth do we need Sqrt? Especially considering that we don’t wrap everything because it is impossible to wrap everything. So Urho is already inconsistent. The moment user wants to use anything from standard library, consistency is gone.

So what if we return vector instead of Vector? We already return bool and not Bool, and we use initializer_list instead of… whatever.

Thanks! That’s really good point, I knew I missed something.

I tried it with ints.
I found one specific case when Urho wins.

Trivial_IntHashMap_EA       1538931 ns      1568775 ns          498
Trivial_IntHashMap_Urho      769266 ns       784738 ns          896

Small hash maps (e.g. 1000 ints) are qucker to fill due to allocator in Urho hash map.
I may try to write EASTL allocator to mimic this behavior, but I don’t wanna do it now.

In the rest of cases Urho is either same or slower.

Changing the coding style together with switching to a container library would remedy this while upholding consistency.

@Eugene Could you include std containers in your tests?

You know, I don’t oppose the idea of snake case itself, since it’s native code style of c++. Given choice, I would have probably started new library in this style.

But the idea of global style change in Urho is unviable due to many reasons I don’t even care to mention.

About std containers, since you all keep mentioning them.

You see, I included std and EASTL in the same vote option for a reason. In order to switch to std, you have to switch to EASTL first, and then switch even deeper. I’m not kidding. EASTL is exactly between Urho Containers and std so we will have to pass this point anyway.

So we can postpone all std-related questions until we completely switch to EASTL.

Could you clarify this? Intuitively it just sounds like an extra leap.

Same. I would LOVE to have snake_case codebase. At the same time it is a crazy undertaking. Not only because it is huge, but also because it is very error-prone. I have been there already. Switching to EASTL was not exactly easy. IDE could help only so much when renaming things. There always are code sections disabled by preprocessor where refactoring does not work. And when it works it isnt perfect. So who is going to do it?

I do not know, but it seems unlikely your stance increases the chances of it happening.

Urho Containers doesn’t have standard interface in any way. They have custom API names and style (CamelCase), custom features that don’t really match std.

EASTL has standart API names and style (snake_case), and custom functionality (additional helper functions in containers and CamelCase bridges).

std is, obviously, standard.

So,

Urho: custom API, custom features;
EASTL: standard API, custom features;
std: standard API, standard features;

Every transition from “custom” to “standard” is manual and separate work in codebase.
So, transition from Urho to std is perfectly split into two phases:

1. Transition from Urho to EASTL, old interface is deprecated (partially removed and partially supported by adapters);
2. Transition from EASTL to std, old interface is removed.

Oh sure. These are totally not related. Not at all. My bad.
Just a convenient coincidente you need Urho Containers out of the way to get the road clear to ***.
*** will totally not take over once EASTL is in.

@adhoc99 Although I understand your suspicion, please try to keep clear mind.

Of course they are related!

Can you understand the concepts of “reason” and “consequence”?

I switched to *** BECAUSE I missed propper container library in Urho.
I want to merge some changes from *** (like EASTL) BECAUSE I want to have decent container library in Urho.

Yeah, but the timing and coincidence are very hard to ignore.

Just because people acknowledge reality does not mean they all perceive it the same way.

@adhoc99 you have some VERY strong opinions for a nobody. I mean i do not recall ever seeing you posting anything. Of course i could be wrong, but your profile is conveniently private so we can not check. Seems to me like someone is afraid to speak their mind using their real identity and is hiding behind a dummy account. You do sound awfully a lot like @Modanung.

What part of “Hello, this is my first time using the forum although I have been using Urho3D for about 2 or 3 years now.” is hard to understand?

Yeah, sure, let me dox myself, that is a great idea.

I’m not Modanung. Modanung has the patience of Saint and he have been here for ages. You should really apologize to him for saying that.

@adhoc99 Do you have github, gitlab profile or something like this?

Sure, but that would dox me.
Sorry, not going to happen.

@Modanung can you please transfer posts of @adhoc99, my responses to them, and your response in the middle, to *** thread?
And discussion in the tail too.
*** thread is already wasted, so let it be offtopic, but this one I want to keep on topic.

@adhoc99 I do not mind anonymity in the Internet, just your IP looks weird tor-exit.r1.darknet.dev

1vanK, that is because I am using https://www.torproject.org/
And that is exactly to protect my privacy.

Would you guys please shut up about adhoc99 and get back to topic. He’s obviously not Modanung, any fool who’s been following this forum for any time at all can tell that. His emotional tones and cadence of speech are completely different. And no, I have no reason to believe that Modanung is some kind of sock puppet actor doing a great job playing multiple roles. Get over yourselves and shut up about the personality tiffs already.

Asking for a monolithic port of EASTL and the removal of currently working code, is foolish. People who actually care about this, should be devising incremental strategies to gradually introduce EASTL into Urho3D. Triggered by appropriate build switches, for those who actually want it. Until such a time as it is feature complete and well tested as compared to currently working code.

This is like Software Engineering 101, or at least Open Source 101. So who of you is going to actually start doing this?

Maybe some of you don’t actually want to do this, you want the blessing for monolithic churn. I personally don’t think anyone should be giving that! YMMV.

Long boring merges like this are not exciting. Maybe the political angle is to create enough excitement, to get a *** developer to actually do something. Or convince a Urho3D developer to do something. But that doesn’t actually matter past this moment, because what is proposed is a marathon. You either have the discipline to run a marathon, or you don’t. You are best off being honest with yourselves about this now, instead of starting something you won’t finish.

Pointing fingers at other people about what you want to have happen, isn’t how open source gets done. Those who wish to champion the EASTL cause, get to work already. You dig into the codebases, you find something you can refactor, you provide a build flag for it. It’s boring and it’s not rocket science. It doesn’t demand discussion.

Actually providing a minor build feature somewhere, is also proof-of-concept that you’re serious, that you intend to stay the course.

If you eyeball all of that, and decide it’s too much work in the real world, that’s fine. Happens to me looking at open source projects all the time.

Sorry, what?.. Did you even read the starting post?
The work is already done. 100% for ***, therefore 80-95% for Urho.

Can you elaborate how do you see build switch for containers?

This thing for every single line of code using containers?

#if URHO3D_EASTL
const auto first = pair.first;
#else
const auto first = pair.first_;
#endif

BS. When you have proof of concept in Urho3D with a build switch, and the limited case test works side by side with extant Urho3D code, I’ll believe you.

If you think it’s trivial to get this cooked up and demonstrated, great! Have at it already. Stop discussing, start showing.

*** is proof of concept Urho3D with EASTL containers.
If you disagree, prove me wrong.

Go open *** samples. They are exactly the same as Urho samples and they run on EASTL containers.

Unless you propose any requirements on how said build switch should behave, it will not happen.
You want build switch? Do something for it. For example, explain requirements. What exactly do you want from said switch.

On the one hand, I do agree that a sudden and complete jump to EASTL may be inappropriate (unless we timed it to coincide with the start of Urho’s 2.0 work), I do feel the push for a build option enabling it is a pretty unreasonable request, as Eugene’s #ifdef example illustrates. If your proposal is instead we wrap the EASTL classes for the most part so that we can still call, for example, Vector::Front(), with some exceptions for HashMap’s key value pairs, I think that might be doable (though I still don’t know if that could be enabled with a build option - unless we change a couple features of the Urho containers like switching to first instead of first_). Then again, I haven’t looked much into how this would work, so others with more knowledge may be able to provide better insight.

Honestly though I don’t care too much about what version of a hash map and the like we use. There are a few changes I do want, however. Firstly, PODVector is still slightly broken, and it would be nice for Urho to not have subtle bugs that will corrupt the heap. Stepping through hundreds of graphics driver memory ‘issues’ (I have no idea if they actually are bugs or not) to get to the point where my program actually wrote to the wrong address to learn that it was because you couldn’t always Push a PODVector to itself was an awful experience.

Secondly, to ease the process of automatically generating script bindings without having to add a large number of annotations, I want every instance possible of function(T* array, unsigned size) to be replaced and/or supplemented by function (span<T> array). (Replacement is my preference, but to avoid removing the existing API I would be content with simply adding the span function and then some additional work can be done to ignore the pointer,size overload)

If there is not strong opposition to the second, I will go ahead and start the work of adding span support to the library (and yes, I do plan on using EASTL for this). That would, I believe, meet your idea of gradually adding EASTL to Urho. I would like other’s feedback on this first, though, as I don’t like having all these pull requests lying around for features I’ve wanted that others were lukewarm to (particularly since the only thing I would directly gain from this is that my very, very side project of automatic script bindings might get easier).

Do you mean you will use EASTL span as it is? In this case, this would be awesome.

This is exactly what adaptors are. I didn’t test them on real code tho.

Frankly speaking: accepting EASTL is bare minimum required in order to get a chance for future. Without it merging anything from *** will be too hard and nobody will step up to do that. Do we need it? Not even remotely. I am fine doing merges from upstream. I do them often and there is so little upstream activity that huge divergence is not a problem. Urho3D needs this however. I already did all the hard migration work and @Eugene did astounding job to ease transition by implementing container adapters. It does not get better than that. This is your chance at maybe having a future. Take it or leave it. I dont think anyone else is coming to offer something like this.

Man, this is so ridiculous.

Please i would like to hear why. Project is stagnating. Nobody is stepping up. Most developed part of the engine by core devlopers is… build system. I do not see a way forward with no contributors. If you do please tell me. I know i can be wrong and i am ready to admit it.

And I wonder why *** has no forum to keep you busy, since it is doing so great.

You are absolutely right, it makes no sense to argue that the project is stagnating. But let’s say we close the Urho3D project and rename *** to Urho 2.0. Will the number of developers increase? On my own, I can say that I just stay with the old version on my HDD, simply because there is functionality that I need. But in the new project I will not take part in the same way as I do not take part in the *** project.

Does getting updates from downstream count as increase of developers? Current state of things surely does not help developer retention, let alone increase. Urho3D lost me, @Eugene and now @Sinoid. This isnt exactly a healthy project growth.

That is, the whole thing is only in the name of the project? You will continue to work on *** just want it to be called Urho3d. And where to go to Urho3D users?

I do not want any users. I never intended to compete with upstream. All we wanted is to share some improvements we have made. Be friendly and contribute, you know. I do not want commit rights to urho repository. I do not want to deal with a community (especially this one). I am interested in code without drama. Some people somehow got the idea that we are looking to take over urho3d while @Modanung fishing for being project leader whose sole contribution is words, and not exactly kind ones towards some. We reiterated multiple times: all we want is to share. Do not want it - fine by me. Less work for @Eugene because after all of this i am not moving my finger to better this community in any way. We could have had a civil discussion that ends with “… but we respectfully decline because of technical reasons X Y Z”. Instead very first reply to @Eugene’s suggestion was calling my bloody sweat a “filth”. This is just disgraceful. No good deed goes unpunished.

You wish for *** to be useless?
Like a clockwork statue.

Nice try.

By the way im sorry for saying you are behind adhoc99 account. Now it seems to me it is pretty clear that you are not and i hope i misread your personality and you would never even think of pulling that stunt.

@adhoc99 would you kindly not use the word “we” as if you speak for the community. I am not someone who uses *** and I am someone who uses AngelScript, so I have my doubts about accepting everything the *** users have offered, but I am supportive of things that would almost certainly improve Urho like a more complete and compatible set of containers. I am clearly a member of the community who’s been here longer than you, and I would appreciate it if you didn’t try to put words in my mouth.

From the heated discussion that has been going for few days I’m really getting a bit confused.

Let me ask in all sincerity and not to offend anyone, if the reason *** was forked was for there to be a better alternative to the vanilla urho3d then why does there seem to be this aggressive push for urho3d to be like ***. If the devs and users of *** are cool with *** then why do they have to “rub it on our face” that *** is better than urho3d and/or that we should upgrade and be like them. Why can’t they just go ahead using it in peace.

It is particularly disappointing that there is a barrage of vitriol, negative and condescending statements coming from the *** team against does that do not want to tow their path. If they really claim they care about Urho3d, observing from how its really been a messy debate, I really doubt them.

I am for progress in all of its form, but what I see going on here is either dominance assertion or a mockery campaign aimed at ridiculing users of Urho3d.

TL;DR: It went weird way.
Let this post be post-mortem of this event.

If you read starting posts of both threads, they are both technical proposals with listed benefits and disadvantages. Proposals from third party that has no authority to push these proposals without consent of Urho mainainers (Note: even if I tecnhically can merge my own PRs, I cannot do it without review and approval of other maintainers).

These proposals were met with accepting attitude from one part of community, conservative scepticism of another part of communtiy, and outright aggression from the third part, which was the most unpleasant part.

You see, when I spend hours writing a proposal, I expect it do be discussed, and accepted or declined. Apparently, certain part of community is incapable of discussion and prefer insults.

I’m not going to speak here for @rku, but I did my best to refrain from unfounded or exagerrated statements.
If you remember any, point me out so I can revise them with cooler head.
By the way, what do you call “*** team”? @rku and me or whole pro-*** side of the argument?

If you look at threads as a whole, you will see that the most of the mess was around single person thowing shit on the fan due to absense of moderation.

I agree with you on this, with one comment.
This event started as technical proposal, and the outcome is what communtiy made of it.

Here is an idea!
For the people who wanted to make Urho3d better.

1. Provide a summary in dot point on what they can do to this engine in terms of features, optimisation.
2. What is their plan on how to add these things.
3. What are expected to be broken if these things gets added.
4. How are you going to draw more developer and user to this engine?
5. Provide examples where the new changes or features that are better than the current Urho3d.

Assuming this is a feature by feature thing (as opposed to a more personal thing - X, you accepted to work on the engine, Y, you are not), how about this:

1. Add relative path support to the engine. This allows the grouping of resources more easily by map section or character, so that within, for example, “Map/World1/Level1/node.xml” resources can be identified as “./tree1.mdl”, “./tree1leaves.png” without having to write out the full path for every resource. It also makes some situations of getting resources from the code simpler (I want the clothes.png file from the same directory as this character specification file, for example).
2. See 6 below
3. Existing resource loading will be changed (all of the GetResource type functions), requiring an additional parameter (a base path) or switching the String resourceName (i.e. path) parameters to a separate Path class so that the relative paths can be resolved. The extra parameter will be inserted before the bool argument as it should be grouped with the actual resource path requested rather than whether an event is sent on failure. It will have a default empty string parameter, so only the case of specifically requesting an event not be sent on failure will existing user code need to be changed (and possibly also custom resources will need to be modified).
4. That’s really not my job, but an engine that is easier to use is certainly not worse. Though you could argue that I have drawn at least one more user to the engine since my friend is now using it for his own project.
5. See 1. In terms of a cost-benefit, we gain support for relative paths at the cost of a slight increase in complexity in searching for a resource, which is likely not a bottleneck in anyone’s programs, so this should be fine.
6. I submitted a Pull Request for this addition two and a half years ago that is still open. Granted, since then I have let it grow out of date with Master so it needs an update to work again, but no one has really indicated support for or against it for over two years, so why would I bother updating the online branch when no one seems interested. I still use it, but I’d have to extract it from my other changes to the engine, which wouldn’t be hard, but I’d just be maintaining a PR that no one seems to want or explicitly not want.

Edit: A new thread has been created to discuss the above proposal. Please direct further comments about relative paths to Adding Relative Paths to Resource Loading

I wrote the above mostly as an illustration of why I think Eugene has asked these questions on the forum. Except for very minor bug fixes (which are almost always very shortly accepted if the coding standards were followed), Urho hasn’t always been too favorable towards adding functionality. It’s not that it’s been particularly unfavorable either, but I can’t say that there’s a small chance one’s work will just sit unused or be outright rejected for even relatively minor changes like adding relative path support to resource cache loading (which does not break existing functionality, and could be made to not alter existing APIs as well).

Certainly, I don’t think your questions are unreasonable, but at the same time how is answering them going to help solve anything?

That aside, here’s another set of answers specifically for the containers question

1. Replace existing Urho containers with EASTL containers. These provide the same guarantee as Urho’s container about consistent cross-platform behavior and size, can also be configured to assert instead of throwing exceptions, and can be modified/extended for Urho as a single set of code as opposed to the highly platform-specific code of the STL. In addition, they add additional types of containers Urho does not have out of the box (and that few are likely willing to reinvent just so Urho can have one as well given the many opensource ones available that would save said developer’s time). One such example is span, which would allow us to replace (or extend) our current interfaces that are along the lines of T* array, unsigned size to a (hopefully) less error prone form that should also be easier to automate script bindings.
2. This work has mostly away been completed by Eugene (and possibly others from ***), though I intend to add the span support myself.
3. HashMap key value pairs will require removing the underscore from first_ and second_ (or potentially very fat wrappers to reference the old names to the new ones). Other minor changes may be required, but I am unaware of them at this point.
4. Same as 4 above. I’m still not sure why this question was asked, particularly in this thread.
5. At least one of Urho’s containers is subtly broken in a way not covered by the documentation but only mentioned in some GitHub issues. PODVector cannot safely Push a value from itself into itself, i.e. PODVector<int> v{1,2,3}; v.Push(v.Front()); is unsafe and can corrupt the heap, in this case a very difficult to track down bug since it doesn’t always happen (only when the vector just be resized) and it will simply write a little past the end of the array, so likely no segmentation fault). Likely this bug has existed for years in Urho without anyone noticing until relatively recently. Rather than having to find and fix these bugs ourselves, we can use the work of others and focus on the important issues in the engine like graphics and such, rather than spending hours on yet another implemention of a Vector in c++. Beyond this, we gain many types of containers Urho has no implementation of which, while not a great benefit, is nice to be able to use a single library’s containers rather than mixing and matching Urho’s and another libraries.

This does sound like a nice feature to have.

Ok, with those idea that you have.
What are your plan to execute it? What are the expected timeline… What are the priority?

Is relative path hard to do? Can’t you manually support it? Some game uses package pk2 etc.

My concern is putting too many things up without a clear plan, schedule and task allocation is not how things should be operated. I can come up with 20 things, but that won’t help much.

Have Urho ever had clear plan, schedule and task allocation in the past?
Don’t really remember it.

Maybe I will be obvious here, but I think I understand the core of the argument.
It is about prioritizing. Preservation before development or development before preservation.

One thinks we should keep what we have first, because otherwise we will harm and chase away exising users.

Another thinks we should develop even if it means reasonable losses in functionality and/or compatibility, because it is pointless to preserve something outdated and outmatched (it’s not about Urho as a whole, but about certain aspects of it).

First group considers this event as an attempt to enforce *** politics onto Urho without care about existing users.
Second group considers this event as an attempt to break stagnation of the engine they like and use.

aaa: Why do you have to break code that works?
bbb: Why don’t you stay on stable version if you are fine with current state of things?
aaa: Why don’t you just move to fork and change it as you wish?
bbb: Why don’t you move to fork and keep it unchanged?

As you can see, this argument is very symmetric and very subjective, so I won’t comment it in any way.

Moving forward only makes sense when facing the right direction.

Lasse pretty much confirmed that Urho doesn’t have and never had any direction.
Therefore, no matter what direction is choosen, there will be ones who disagree with said direction (=doesn’t consider it “right”).

Current Urho3d is feature complete to me. It only need a faster,
pretty, rendering technique.

Any vector with a non-zero magnitude has a direction. Direction is not the same as a 5-year plan that has been set in stone and signed with blood.

It’s good.
What are your reasons to participate in this discussion instead of using current version of Urho from now on without any care about future plans?
We do it with 3rd-party libraries all the time. You are fine with version – you stick with it.

Can you please elaborate? I don’t really understand what do you mean here.

What is a right direction for some people may not be a right direction for others. This means there is no right direction. Nor wrong one. People who write code set the direction. You can write the code and promote direction you see as right. Just like we do.

1. Urho current PBR implementation is slow and not complete.
2. Urho defer render, lightning method is not pretty.
3. We need xxAO variant of render technique support. Your variant of AO looks good.

If you think you can identify the trajectory and milestone for your proposal, you should first come up with a plan. Identify the required tasks and time line. This will help inform people so that they can support your topic.

It seems to me that most of the posts in this thread are individual targeting and bashing without much constructive material. This maybe due to lack of persuasive approach and plan given.

I know that you and some others have been here long enough and contributed to some area in the engine, which is great. But I do think some people really act like they are god, which is really getting out of hand

Anyway, I’m supporting any changes and bug fixes that help this engine. Please give some constructive information to persuade the community.

Thank you for elaborate answer.

I do have certain plans that partially cover your items, but I’m not sure about implementation details. It’s always risky and hard to make changes in Urho renderer.

I’m the type who prefer talking about something after it’s done, not before or in-progress. For example, I’ve posted my lightmap/GI solution the moment I considered it finished and ready to use, not incomplete prototype for show.

I have several ideas how to change renderer, but I will not make any plans or promises until I confirm they will work out well.
I want to try one radical approach, but it will require huge commitment of effort and therefore you will not get any updates in the nearest months about it.

4 posts were merged into an existing topic: Adding Relative Paths to Resource Loading

@Eugene @rku Any pull requests for EASTL coming soon?
Given EASTL compatibility with stl, it’s going to be very easy to switch to stl containers in the future if needed. For now, it looks like a decent choice.

@Modanung We all appreciate what you do, but the engine does lack contributors today, and Eugene and rku are to be appreciated for their willingness to move urho forward. Sometimes a leader has to compromise a little on his vision for the greater good.

@adhoc99 You are very disruptive. You remind me of Leith (if you are not him). A change in your attitude might be beneficial to you too.

You see, I’m not willing to deal with AngelScript bindings.
They may be nice for user, but they are a hindrance for ones who want to make wide-scope changes.
And Urho cannot abandon AS, since about 1/3 of people here (or so I think, according to poll) really want AS.

If you want to get EASTL in Urho, you probably better ask people who are ready to deal with AS whether they want to do such PRs, not me.

Maybe I will eventually try to do such PR, even if it means working with AS, but I’m totally not in the mood to do it now, not after reaction I got. If so many people are reluctant to accept just EASTL, why would I push?.. I still somewhat want to try PR EASTL into Urho, but it’s very low priority task for me.

21 posts were split to a new topic: Script binding automation

@cosar I very much appreciate @Eugene’s presence, but “forward” is a subjective term that - if a project is not to rip itself apart - has to be agreed upon before taking leaps. Part of @rku’s efforts have been directed at actively demotivating engine and game developers alike from continuing to use and contribute to Urho3D. I appreciate his positive contributions, but he has often displayed the opposite.

2 posts were merged into an existing topic: About *** and Urho3D

@Eugene If you don’t want to deal with AS bindings, then just don’t do it. If you are willing to, you can get the branch started and change to EASTL without modifying AS bindings (you can disable AS) and somebody else can step in for helping with the bindings. I guess I can find some time to help if needed.

@Modanung I understand your position, but the truth is that Urho3D had a significant drop in contributions for some time now. I know I probably shouldn’t talk since I don’t contribute, but I do like this engine and I would love to see it moving forward.
One good thing from all this discussions is that there is still a lot of interest to see this engine thrive. Maybe @Eugene and @rku can also pull back a little bit and help getting things from *** one by one with support from the rest of Urho3D community.

As I said, it may happen in the future after I finish my current tasks.
It was my intention in the beginning, as I explained in the very first post, but after received pushback I’m not inspired to change anything in Urho for now.

About EASTL branch… you see, unless AS is dealt with, such branch will never get merged, and I hate wasting time on something that may never get merged. I can start working on EASTL branch when/if I have the confidence I can complete this work on my own.

From a technical “all options on the table” standpoint, this statement is false. Code that does EASTL could be merged, if it leaves AS mechanisms alone. 2 different ways would have to be supported, for quite some time. “Merge” doesn’t mean “I take over with my way, you have to lose your way.” Migration strategies, in general, do not have to be all-or-nothing propositions. They can require end users to make choices, like what feature set they’re going to utilize, and making people rewrite some code if they want to migrate. Providing multiple options does increase complexity and work, especially for test coverage. In software engineering, that is generally the cost of keeping working code working.

From a personal energy standpoint, it is of course true that you will work on, whatever you actually want to work on. If you don’t have the software engineering appetite for doing a long term incremental migration to EASTL, and really only would ever choose to implement a simpler, all-or-nothing, disruptive approach, that’s your choice.

I just hope we’re all clear on what is possible. The tradeoff is work.

People look at any given *** “feature” or “offer” in exactly the same way. What people will actually do, depends on what people think is important to actually build towards.

If some class is moved to EASTL, then interface of this class is likely to change too.
If interface changes, bindings have to be updated in order to build scripts.
If bindings are not updated, then PR cannot be merged due to broken bindings.
Therefore, without changing bindings PR cannot be finished.
Please point out what exactly statement here is false.

You keep talking about incremental migration, but you never offered a solution regarding how migrated modules shall interact with not migrated modules without redundant overhead.

Unless you have something in mind, please refrain from suggesting unviable strategies.
I have no clue how to incrementally migrate A without touching B:

class A
{
	Vector<int> GetNumbers();
};

class B
{
	void Update(A* a)
	{
		numbers_ = a->GetNumbers();
	};
	Vector<int> numbers_;
};

Write a 2nd one. Or subclass it. FFS why do you think C++ and OO got popular in the 1st place? Why do you always talk about moving things? Why not adding things?

This sounds like some managerial policy BS that could be addressed if someone had a reason to do so. Like if additive EASTL insertions were valued.

Of course I didn’t. It’s all details, down in the weeds. I’ve got my sleeves rolled up for other kinds of game development right now. You just keep talking categorically like these sorts of things can’t be addressed in projects, and they can. Long Ago, I got paid real money to work on CMake build systems, for Mozilla. I can hum a few bars at software engineering and migration. This isn’t rocket science stuff. It’s a question of will, and realism about how much work something entails.

I get it if you don’t want to do migration and build systems “flip a switch” stuff. I got good at that sort of thing, because of all the people in Open Source Land who don’t relish doing all that stuff. Leaving me to figure out how to even build their !#!# broken ass software. And it was ultimately a career ender, I got real good at something I hated.

What is your problem with that? Pay a cost already. SW ENGR 101.

Did you ever considered that non-incremental “all-together” migration may cost by the order of magnitude less effort than incremental?
Writing a second interface is a lot of work. And only a fraction of classes may be subclassed in Urho.

It’s very easy to be theorethical and speak about abstract changes in abstract projects and how they should be done in perfect world.
Sometimes the world is not perfect and abstract solutions don’t work here as well as in dreamland.
For example, you proposed abstract solution may cost 10-100 times more than solution already implemented, de-facto making whole migration thing unviable.

Did you really just suggested to introduce redundant performance overhead in real-time game engine?

If we go incremental, we introduce performance loss at the joints between old and new containers.
And performance loss is something I care about, especially in real-time game engine.

Yes. You clearly think like a bog standard 3d optimization jock, not a project manager concerned with multiple generations of user support. You wouldn’t believe the amount of stuff that exists in CMake to support backwards compatibility for build systems. You dismiss engineering solutions that are obvious to other people in the computer industry because they are not perfect, speed of light code. Get over that, and you can actually solve the political issue with Urho3D and *** development. If you want to.

The problem is you like EASTL in the 1st place because you believe in speed of light code and don’t want to ever compromise.

Here is your political framework at the build system level:
USE_URHO_CONTAINER - no new code
USE_EASTL_CONTAINER - only new EASTL code, breaks AS and anything else in your way.
USE_BOTH_CONTAINER = USE_URHO_CONTAINER & USE_EASTL_CONTAINER - the most difficult mode that makes both work simultaneously. It can sacrifice some performance.

Your political goal is to get people to move between the 3 categories. Until such a time most people don’t value the old stuff anymore and it can be gotten rid of. That will be years from now.

Great.
So, user had his nice function in the engine API that returned String by reference and has literally zero cost.
User used this function a lot in the hotpath.
Then we do migration, and now this function always return copy of String because there is no more String because underlying container was migrated to ea::string.
This may literally nuke whole application performance.
And you are saying this is totally fine because nobody should expect real-time game engine to work fast. Right?

There is no new code and no old code. There is just one copy of code, with very high level of entanglement and a lot of containers.

I don’t want to implement detached from reality suggestions from a person who has no clue about the task. If you used some approach to solve your tasks in the past, it doesn’t mean that the same approach works on all tasks in the world.

I understand the concepts of migration and legacy compatibility and how it should be done, and I employ this approach more often than not.
I also see no proof that your proposed approach is even remotely viable in this specific task.
Past experience with completely different tasks is not a proof. Neither authority is.

I’m ready to compromise and I’m concerned about legacy compatibility.
I also understand physical limitations and understand that sometimes it is de-facto impossible to upgrade things perfectly.

It’s all details, down in the weeds

I’ve seen a lot of project managers (it was a lie, I think I saw only one or two) who forget that the code is the core. And architectural or management decision is worthless if it doesn’t take into consideration the code it is going to be implemented in.

I’ve spent a lot of time thinking how to make container migration as smooth as possible, and I haven’t found acceptable solution. And here you are, saying that it’s totally doable and giving literally zero actual practical advice. What gives you a reason to think that person who never worked on the task knows more than a person who spent a lot of time doing said task?.. I’m not a project manager and I’m not an architect, but I worked on the architecture and I’m not an idiot.

You asked why. The answer is you’re basically biased towards your favorite piece of tech, EASTL. You don’t see the reality of other people wanting their stability. Like a lot of open source developers, it’s not that important to you. Nobody’s paying you to make it important, so you don’t look for a win-win solution.

You aren’t thinking in terms of phased migration, either in your own work, or in other people’s work. Developers can make choices about whether string functions are critical to them. They can rewrite some code as they migrate, if the new regime is bringing enough benefits in other areas. To worry about whether USE_BOTH_CONTAINER is suboptimal, is to completely miss the point that it’s a transitional framework. If people really care about the performance of something, they’re going to either go forwards to the EASTL and no AS approach, or stay back on the USE_URHO_CONTAINER approach. Your goal should be to get people to migrate forwards. If you’re serious about transitions, and implementing “depreciations”, like big projects like CMake or OpenGL actually do in the real world.

What do I get out of participating in this discussion? I’ve been watching Urho3D for a long time. I know the project leadership and more importantly project cooperation is in trouble.

I’m hoping, that you are clever enough to see what I’m telling you, so that I don’t personally have to show you how to implement EASTL without harming AS. Because if I have to do that, then you haven’t contributed any development value to me. In fact it would pull me deep into the weeds for an issue I’m not that vested in, “what the perfect container class is, and how that can contribute to other *** migrations”. But I’ve been around the block in Open Source often enough to know what the technical and political dimensions of the problem are. It’s a shame to see a concept fail, for lack of know-how.

I can help you with the CMake build system end of the problem. I don’t care what Weitjong has cooked up, I’m quite sure I can implement whatever he thinks is The Right Way To Do It.

The real question, to me, is whether you actually want to change the DNA of Urho3D over time, or you just want to do everything your way because that’s easy for you. It’s not easy for everyone else which is why such generous offers in open source are respectfully declined. Making everyone happy is work. A price must be paid somewhere, either in designing interfaces, or performance, or multiple transitional steps over time. A user might ultimately have to write 3 versions of code to make a final transition to your new regime. If they can do that gradually, and keep things working without suffering too much performance loss to get there, it’s a win.

I am, and the current state of affairs (migration all at once, optional container adapters) is already the best and the smoothest way to migrate I managed to find. You (person who never touched the task) keep saying there is a better way, while I (person who spend a lot of time on this task) don’t see any. It’s not just me being stubborn, you know.

It is technically impossible to make existing Urho code work with both old and new containers without significant decrease in code quality, maintainability and/or critical impact on performance.
If you think otherwise, please provide arguments, not baseless theoretical statements detached from reality.
If you think it’s possible, prove it. Explain on real code (e.g. Node), not on imaginary examples.

If you don’t have time or willingness to do so, that’s totally fine, you don’t owe us anything. Just remember that suggestions from a person who never worked on the task have very little relevance. It’s too easy to talk about perfect solutions when you have no codebase limiting your fantasies.

@bvanevery While I don’t doubt your experience in related issues in other projects, I’m going to have to take Eugene’s side in this. Even if someone were to make a PR in which support were added for both EASTL and the Urho containers, presumably through a large number of #ifdef's, I would actually recommend against accepting that into the Urho codebase. It seems pointless to me to have twice the number of functions (and five+ times the number of lines) in header files alone for everything that uses a container. Simplicity is a good thing, and this seems like trying to make many times the present work that would be required simply for the sake of people not having to refactor their code to use first instead of first_.

Personally, I think this is only sometimes true. Certainly, it could be (dividing breaking changes into minor refactoring sessions seems like a pretty decent approach, overall), but what if, following your own setup, it’s something like user has to spend 1 hour refactoring now vs half an hour every few months. I think then the win is actually with just getting it over with quickly - here’s where you were, here’s a clear list of instructions for how to port to the new containers, it should take an hour or two, have at it (rather than dragging it out marking features as deprecated and removing them slowly piece by piece (so user’s code ends up broken several times over rather than once).

Also, others may disagree, but I don’t see how doing many times the work so others can make all the same changes (just later rather than sooner) is a win for Eugene. It’s a win-win for the paid developers because they get paid to make the other people win, but I don’t really see how Eugene wins in this situation (nor, for that matter, how we add the community win other than that someone is volunteering their effort to bring an improved container library to Urho)

That’s not too say I don’t agree with some of your ideas. By all means, let’s go for a gradual transition, provided it’s only going to be a bit more work and not like ten times as much. I would like some clarification as to how you think cmake solves the problem, though - wouldn’t that literally just be adding a cmake option that adds a preprocessor define? (Which, speaking from personal experience, is really straightforward with our build system - just copying the existing cmake code and changing some names). Perhaps I’m just missing something, though. I have very limited knowledge of cmake.

@Eugene, do you know what features are missing from these? If it’s not too many (i.e. unless that actually makes the classes useless) I say we add an anonymous union to Pair<> between first and first_, etc., and mark the underscore variant as deprecated, as well as all the other functions not supported by the adapters. Then in a few months from now, or perhaps with version 2.0, we actually switch to early EASTL (as long as someone deals with the AngelScript bindings). Thoughts?

At this point, I believe Eugene lacks the will to steward this transition.

I am insufficiently invested in the core issues, “EASTL --> migrating *** features”, to provide a solution for him. I will bow out from any further discussion of *** related stuff. AFAIAC, it’s all DOA. My opinion after reading many posts now.

I will save my Urho3D potential energy for my own, currently undisclosed core concerns. My own issue is, they tend to run counter to a lot of publicly expressed concerns. Worst case, Urho3D has no value to me. But I’m on the cusp of either writing my own 3D engine or trying to make Urho3D work.

I believe open source is about one’s will, and not about squabbling with any developer over the details. Like whether something is “advised” or not, or will be “more work for someone”. People who want something to happen, go make it happen. Then others follow in their wake, because they actually did the work, and it works. That is how open source projects actually get changed. It may not be Open Source 101, but it’s no more than a 300 level course in What Is To Be Done.

When viewing the *** repository, I am increasingly convinced that switching to EASTL containers is simply a change for changes. There is nothing ideal in them and they have to be supplemented with the necessary functionality. In theory, we probably get some kind of speed increase (which is debatable, since no one checked the difference in real projects) and get an ugly syntax that stands out from the general style. Just as it is necessary to add functionality to EASTL, we can fix / modify / improve Urho3D containers.

p.s. That’s not cool https://github.com/***/***/blob/master/Source/Urho3D/Container/Str.cpp

Or, in other words, you can only talking about abstract ideas and have no clue how to implement your own suggestions.

I wonder if you ever did transition as pervasive as container switch in project with strict performance requirements.

At this point, I believe that you understand only a fraction of issues that arise in this task. You are addressing only the issue of compatibility, completely ignoring the rest of issues related to code quality and performance in critical paths.

When several Urho developers tell you that your suggestions are unviable, it means something, you know. I believe that any Urho contributor will tell you the same.

Are you ready to invest hundreds of hours into Urho containers so they get on the level of standard library? I wasn’t, and so I switched.
Standard library exists for a reason.

This is syntax of standard library. You cannot get rid of it.
Wrapping every single name in CamelCase for sake of consistency is weird path to walk. I prefer making interesting things instead of reimplementing things already done.

Are you ready to invest hundreds of hours into Urho containers so they get on the level of standard library

The fact is that I have never experienced any problems with standard containers. In the end, you can use any containers in your projects, why change the containers that the engine uses?

Because I don’t only write code for my projects, I write code for the engine. Also, using multiple container libraries require ugly transition layer in between.

You have just admitted that you cannot properly utilize C++ standard library.
Because a person who regularly uses C++ to full extent cannot not complain about non-standard container library missing features.

For example, if you are good C developer and used to write code this way, you will see no benefit in migration from C++98 to C++11. They will both look the same for you. However, for a person who used to use C++11, such migration will be critical, because on C++98 they will feel crippled.

Same for Urho containers. If you learned 5% of standard library and that’s enough for you, Urho Containers will be fine for you too. If a person learned 90% of standard library, Urho containers are quite annoying to use.

Can you show your projects?

Define “projects”.
Also, I don’t get how this question is related to topic.
Considering that I just said I don’t need EASTL in Urho for projects. I need it for the engine itself.

Define “projects”.

…

Considering that I just said I don’t need EASTL in Urho for projects. I need it for the engine itself.

Is Urho3D works 5% of opportunities, but *** 100%?

I have published my project few weeks ago and you have seen it. You either have terrible memory or different definition of the word “project”

And yes, if we are speaking about fine tuning and optimization capabilities, it is 5% vs 100%. With Urho containers you have no options, only default vector on heap, default string on heap and default hash table on heap. And no algorithms, except sorting.

As an inexperienced user, I understand best of all through examples. What restrictions have you encountered?

The simplest example is removing heap allocations on hotpaths. This is simplest optimization that can make things several times faster. In EASTL you can change one line and get rid of heap allocation, without changing behavior of the rest of the code. Then you can measure if it’s beneficial or not.
Good luck trying it with urho containers /wink/

By examples, I meant pieces of code from your project.

In EASTL you can change one line and get rid of heap allocation, without changing behavior of the rest of the code.

I do not fully understand. Did you create vectors on the stack?

I cannot link you from phone, but new serialization of Serializable in *** uses what I mentioned to achieve guarantees missing in legacy serialization without introducing heap alloc on each serialization.

Yes, EASTL can use stack to store small containers and fallback to heap if overflow.

What is not cool there?

External functions to add missing functionality

I agree it looks more nasty. However, these functions specifically have quite limited usage. Have you ever needed to use them? I haven’t. You need to call them once in text renderer, and that’s it. They also have more generic signatures, than functions in Urho containers.
Maybe we should just move them all to string utils. We have a lot of helper functions there.

String having two length methods returning different things is not exactly better. Even more so considering that string is encoding-agnostic byte dump. If we happen to have to work with strings using different encoding some methods are flat out illegal/invalid. So free-standing functions do seem a lesser evil.

I actually encountered some string related problems when writing my RichText component…I have this:

PODVector<unsigned> unicodeText;
for (unsigned i = 0; i < text.Length();)
    unicodeText.Push(text.NextUTF8Char(i));
...
for (unsigned i = 0; i < unicodeText.Size(); ++i) {
    const FontGlyph* glyph = font_face_->GetGlyph(unicodeText[i]);

Not a big deal tho, it works flawless. If I start doing big O notations (which I’m not good at) I’ll get in trouble

Even Utf-32 without normalization does not guarantee that one character will be represented by a single code point.

That’s false. I know exactly how I’d get in there, roll up my sleeves, and make it happen. I also know how many months of work it would take to iterate and test it. It’s not Low Hanging Fruit. That’s the whole reason I took you to task several threads ago, when you said you had proof of concept for EASTL in Urho3D. I said you most certainly didn’t have proof of concept for migrating *** features into Urho3D main repository. We at least managed to get on the same page about that, about the non-triviality of the job.

I wonder how many more years of development experience I have than you do. I’m a 50 year old fart. You think nobody ever bravely waded into spaghetti before?

it means that a few people who are paying attention, are unwilling to wrestle the bear.

So far, I don’t value anything that *** is offering. You personally are not offering any exceptional code merging skills. I don’t personally value C# in any way. Although I don’t use AS, I don’t want to see it undermined. Whatever working ecology Urho3D has, needs to be kept. Of the issues that have been discussed, I find the archaic tolua++ to be interesting and will be looking into that. I don’t believe in PBR, I’m using crufty 11 year old business class laptops with minimal DirectX 10.1 HW feature sets. Gameplay is king for me, and what I actually want out of a game engine, is stuff that allows me to rapidly prototype games. I don’t believe in 3d modeling and animation packages, not even Blender. As a lone wolf indie I’m not going to waste my time on producing “shiny” PBR art assets.

In saying anything at all, I was probing your willingness to provide stuff of value to others. You’re not seeing the kind of merge framework, necessary to make you happy and others happy. Maybe in coming months, if I actually go with Urho3D, I’ll write the build / merge framework and just tell you one day “here it is”. Then see if you’re actually ready to put EASTL or whatever else in it.

Probably a lot, since I’m in the beginning of this path.
And that’s why your position is so surprising for me.

You are valuing smooth transition thru depreciation above everything else and I understand why this is important.
I don’t understand why you ignore the rest of aspects.

There is compatibility, so users from old version can smoothly transit to the new one.
There is code quality, so we can keep core code clean and simple.
There is real-time performance, so we don’t turn Urho into yet another bloatware.

Sometimes you cannot get all these things together. And sometimes it’s better to sacrifice compatibility than to ruin code quality or performance.

Big projects undergo breaking changes from time to time. That’s why we have versions, you know?

And I agree with you that *** has almost nothing to offer… now.
I’m just afraid that if this change at some point, it will be too late to merge features back into Urho.

This is not something to start with. You yourself saw how many people came to the project, added the code and left. Now we will break what already exists and works. But there is no guarantee that something will be added.

Hence USE_BOTH_CONTAINER. It’s a transitional space where performance imperfections are allowed.

They shouldn’t. Open source projects often die doing this sort of thing. I’ve been around long enough to see it happen more than once. You can’t just throw out committed users and expect that to be consequence free.

What these many threads have taught me, is that Urho3D is sufficiently far along in maturity and developer energies pulling in different ways, that a framework for getting people to cooperate is necessary. It can’t just be “well my version wins now”. Without a framework for What Is To Be Done, at this scale of development, the kind of long thread “I wanna contribute but I can’t, because, Reasons” conversation will repeat indefinitely. Project won’t be able to absorb new blood, project will die.

It could be that lack of leadership, makes the stress on the cooperative aspect, much more obvious.

I’ve evaluated a fair number of open source 3D engines. If anyone out there had the feature set I wanted, I’d be using it. Ogre was stagnant for a long time. Can’t remember this past year, but don’t remember being interested. I’m not doing Godot’s GDScript. It’s also more 2D and mobile oriented than 3D oriented. Urho3D at least has a sane Lua basis, whatever its other problems. There’s this guy who wrote Wicked Engine, which has the benefit (to me) of being DirectX oriented and not bothering with non-Windows platforms so much. But he’s 1 guy and he’s a PBR jock kind of guy. He’s not going to produce the prototyping development stuff I need, and I bet you trying to add features to some 1 guy’s project, is going to be “stepping on his baby”. Urho3D’s current lack of leadership is actually an advantage in that regard. If I did something substantial, nobody would be likely to stop me.

Not doing Rust either. I have investigated so many programming languages. I hope Jonathan Blow ships his Jai language soon. I hear a few people are testing early betas.

Can you explain how Urho API should look like with Urho containers only, EASTL containers only, and both?
I keep asking this question, but you keep evading it.
And I don’t say you need to actually write code and implementation anything.

Migration to EASTL is code task first and foremost. Any proposal you make is completely worthless unless it considers actual code we are working with.

So far I believe that it is impossible to have such build switch without impacting performance and drastic decrease in code quality, and I’m not talking about transition state only. Am I wrong? You cannot imagine how happy I will be if I am.

Just one simple example. Node::GetChildren(). How shall it look like?
This is one-line function, as simple as possible. Can you implement your proposal for just one line of code, so we can understand? If your proposal is doable, it shouldn’t be hard.

No, no, a thousand times no. You don’t get what in the weeds means. The problem is, this is the only level of development you’re willing to think about. Because it’s tangible. What you’d really have to do to solve the problem, is get into the code for a month and find the answer. By iterating and changing and testing over and over again, until you have the design. You don’t have the design now, I don’t have the design now. You ask me to pop the design off the top of my head, and I keep telling you, sorry it doesn’t work that way. I talk about personal will and you don’t want to hear that. You have to be willing to pour the blood on the table to find the right answer. That’s the only way this sort of non-trivial problem gets solved.

Your huge blind spot is you imagine solutions must be both permanent and performant. Discard those constraints and you have options.

That’s why I keep talking about the metaphor of USE_BOTH_CONTAINER. If you actually get into the problem, you won’t actually be able to implement a global switch like that. You’ll have to phase in whatever particular chunks of engine code you run into. You could end up with 5 build switches for all I know, phasing different things in. You’d have to test every phase you implement, verifying the progress, not breaking anything.

It’ll be work. People can get paid big bucks per hour for this sort of work. I did it for Mozilla once. Sadly I blew the project, but I did get paid at least. I think my “temporary scaffolding” method was actually working, and if I’d only bid the project as 2X more expensive and time consuming, I may have prevailed. But, the budgeted time and money ran out, and I didn’t have it working. So that’s that, haven’t done substantial CMake or gotten paid for anything since. Lessons learned.

Let’s pretend this debate and your question was a consulting contract. Like you’re the “client”. This is the point at which I’d be saying, for $X up front, I do a study for you. Because I know it’s going to be at least 20 hours of work, to tell you the kinds of exact things you want to hear.

This is open source, so of course I don’t do that. But I don’t waste my 20 hours on your need for specificity either. If I have to tackle the problem, you’ll get it six months from now and I won’t talk to you about how to do it. I’ll just say “here it is.”

My 20 hours is going to get blown on tolua++.

You miss the point that I have already spent significant time doing exactly what are you talking about.
I made two attempts to migrate Urho to EASTL smoothly in past years.
I actually worked on design for some time, and only after that I came to conclusion that completely smooth transition is impossible.
Feature branch may be incremental, but the merge should be all at once and it will introduce non functional incompatibilities.

When a person who spent 0 hours on design tells a person who spent 40 hours on design that there is better design without suggesting anything specific… It annoys me, even if you end up being right.

So stop trying to do that. That’s what the 3 categories I gave you are. They could be paraphrased as BEGINNING, MIDDLE, and END.

I can lead a horse to water. I can’t make them drink.

Urho3D active use containers internally (events, sorting batches, etc), so I decided that Samples are a good test for containers.

Urho3D builded with AS, *** without bindings. Both static runtime, release, x64, without tools.
Build time: urho3d - 7 min, *** - 21 min (I think this is due to the large number of templates)

I modifie Urho3D\Source\Samples\Sample.inl to match *** hud

void Sample::CreateConsoleAndDebugHud()
{
    ...
    debugHud->SetMode(DEBUGHUD_SHOW_STATS); // Added
}

I use Fraps for showing FPS becuse Urho3D do not show fps with minimal hud.

Most likely, my testing is wrong because the examples are not identical. However, in most cases, Urho3d renders more triangles and batches.

EDIT:
I think the difference in the number of batches arises from the fact that Urho3D renders characters twice (with a shadow).

Did you count 3rd party libraries, or it is engine-only build?
*** uses more 3rd-parties than vanilla Urho.

If you are testing latest *** version, then comparison is incorrect.
If you want to get more or less fair test, you need to compare version before I added baked GI support in ***.
Also, I would say that the only relevant test is HugeObjectCount. The rest is just too tiny to measure CPU reliably.

Any other tests are welcome

EDIT: How are lightmaps used in the samples?

They are not used, but you still have to pay for an ability to use Global Illumination. I can make it completely optional in build options if I ever get such request.

For Urho I think this would make most sense, with the option being turned off by default - skipping the build of a possible sample - and a clear mention in the docs.

@Eugene *** samples render less batches. Seems like upstream is paying extra cost for something.

This is really weird, they must be the same. Maybe *** doesn’t count UI batches that go thru SystemUI? Do we count UI batches in these stats at all?

I made the assumption above

You must be right. Debug hud and console are rewritten in imgui and batches are indeed not counted as everything is rendered by external renderer.

Is it possible/viable to make Urho work with both STL and EASTL? So that with a single (preferably compile-time) switch you can check both?

That would be a HUGE selling point of the engine. So much so that you could use it as a benchmark for somebody’s new STL implementations.

You mean, both containers in the same API as an option? No, it’s not, unless we make a lot of sacrifices.

Also, I noticed small but noticeable performance drop in HugeObjectCount sample. I still think it’s not important, but I’m revoking my vote for EASTL until I have some tangible improvement tied to EASTL.

I was wondering about a “killer performance demo” for EASTL. From anyone, anywhere, any app. With benchmark proof. EASTL could make a C++ developer’s life “more sane” for all I know. Or it could successfully “outsource development and maintenance”, for all I know. But without a demonstrable performance argument, I don’t see the point.

The point is that I wasted a lot of time poking Urho containers to fix bugs or make them do what I needed. I’m bored of it. I don’t want to fight container library every time I want to experiment with code.
I believe it’s still impossible to put noncopyable classes into Urho HashMap…
And I don’t believe we will ever get SBO that is considered must-have in standard libraries for a reason…

I don’t expect to get free performance just by replacing custom and simple Urho containers with significantly more complicated EASTL containers. This is not the point.
They have almost the same performance (with deviations in both directions), and this is fine for me.

I will try to optimize code using EASTL features, one day. Not today and not tomorrow. We will see how it will end up. I’m curious too.

I understand that UrhoContainer gave you programming pain, and that you wish to alleviate it with EASTL. I’m asking if anyone anywhere has ever demonstrated real performance gain with EASTL. Yes I can web search that, but so far I have not.

I get niceness, stability, completeness, outsourcing arguments (if true). Asking about performance.

Ok finally bothered to search. Claim in EASTL repo is:

It is a fairly extensive and robust implementation of such a library and has an emphasis on high performance above all other considerations.

The source tree contains a benchmark.

On Reddit I’ve so far found references to very old benchmarks, but that makes me realize modern benchmarks are what’s relevant now. One opinion from 6 months ago:

I actually did this test at a previous company. We were primarily concerned about compile time and run time, and sorry I don’t have the full results from it, but EASTL was never better, and often slightly worse in every dimension.

I imagine this wasn’t always the case (I mean, they must have written it for a reason). But today’s STD libraries have gotten a lot of attention and are somewhat coupled with the compiler they’re written for, so it’s not really a surprise.

and the next comment:

In the company I’m at we had EASTL available for our clients and it became such a big cost of maintenance that we made those tests to see if it was worth it. The tests basically revealed that equivalent containers were just as fast in our usages. It took a while to convince everyone that we had to remove. The outcome was to upgrade to c++14 (I do not remember why) and a suprising decrease in compilation time, specifically in one of the bigger projects.

Discussion from Hackernews 2.5 years ago:

You can clone the repo and run the benchmarks yourself. When I last did it it was quite mixed (compared to libc++ on a MBP) - a lot of things were the same or slightly faster in EASTL, occasionally some things were an order of magnitude faster, some things an order of magnitude slower. Those were in what I would consider “edge cases” rather than general usage, for which the two were fairly similar.

As other comments have said though, the main reason it’s used (and the reason I was interested even though I don’t do games) is because the allocation story in the stdlib sucks.

The above is all I can find on the subject, after straining to sift through Reddit, Hackernews, Gamasutra, Gamedev.net, and the web generally. Done with the research now. I conclude that to know about performance, one must benchmark current computers and platforms.

Good research. One crux of standard libraries has always been performance inconsistency across different platforms. No surprise here, different implementations are different. A lot. At least EASTL performs same across all platforms. Your findings however seem to hint that situation is changing. For example some recent MSVC STL did major work on debug performance. And then there are extra bits provided by EASTL that are not in stdlib. Be that as it may, there is little discussion to be had about whether current containers better than any alternative - they arent.

They do not seem bad either.

The same is true for Boost.

Just ignore boost. It is a monster. Too big!

Unlike Boost, EASTL is just a dozen of headers.

They are much harder to deal with when a person is used to use C++ standard library. The more familiar with STD library you are, the more painfil Urho containers usage becomes.

Especially considering that Urho containers cannot be used with standard algorythms. At least, they couldn’t when I tried it years ago.

You mean when they were without begin() or end()?

Nope, I mean standard algorythms like std::sort and so on.
Urho containers cannot be used with such algorythms because they don’t have compatible iterators.
They may work from time to time, but it’s not guaranteed.

I thought Urho3d already has fast/quick sort?

It has sort, although I haven’t measured its performance.
What about remaining 99 standard algorythms? Urho doesn’t have them.

I still prefer migrating to (wrapped) STL instead, and do not see how EASTL would make a convenient stepping stone instead of a detour.

Migration to EASTL is X work. Migration to STL is X+Y work.
Migration to EASTL should be published all at once.
Migration from EASTL to STL can be done incrementally.
What part here you don’t understand?

I think you are forgetting Z.

What exactly is Z?
Build system setup for EASTL?
Well, yeah, it would be extra work.
Sometimes you have to pay in one place to win ten times more in other place

Don’t forget that STL is lagging behind EASTL, too.
Only STL in C++20 can compare to EASTL, and we can not migrate to C++20 in nearest 5-6 years.

Therefore direct migration to STL is not only several times harder than incremental one, but is unviable in the nearest future.

I do not see a problem with that. The code is still in the process of transitioning from “C” to C++11. Sure, features are neat, but maybe we need a spring cleaning first and look at C++14/17? Also with cross-platformness comes backwards compatibility: Moving too fast leaves some behind.

If you wish to migrate Urho to STL for C++11, I don’t want to discourage you. It would be great to use standard library in Urho, even if it’s just c++11 (unless it ruins performance, but only profiling will tell if that’s the case).

I’m only stating that I personally don’t see a point in migration to lagging standard library and don’t wish to spend additional effort on it.

What if for sorting of batches are not needed remaining 99 standard algorythms?

What if I solve more diverse tasks than just sorting of batches?

And Urho containers are positioned as general-purpose containers, not containers-for-sorting-batches.
As far as I know, it is discouraged to use std::vector in main Urho codebase, especially in public API.

For example, I needed to swap first N elements of vector with the tail of the vector, without any reordering and intermediate variables. I don’t want to spend more than 10 seconds on such trivial task.

Why would you do this for a list of nodes?

I needed it for tetrahedral mesh generation. Why nodes?..

Because I can no longer recall the examples of containers that the engine passes to a user from its insides.

Use EASTL with 99 algorithms

Yeah, I used EASTL in this case.
Now is the question. How can I contribute said class into Urho, if Urho policy discourage standard containers in public API and I don’t wish to rewrite it to clumsy Urho containers?

Are you going to copy vertices into urho containers one at a time? If no - memcpy

I cannot comprehend this question.
I have a class that I write with standard containers.
I cannot contribute it to Urho because Urho discourages standard containers.
How “copying” is related to it?

If you do not plan to copy the vertices one at a time, then you need to copy a piece of memory, right? Who cares which container?

I never said anything about copying. Why are you talking about it?

If you do not want to pass an array of vertices to the engine, then what is the question?

I repeated the question two times already.
I want to contribute brand new class into Urho.
I use standard containers in this class, as public members.
Urho policy says that I should use Urho containers, at least in public part of my class.
What should I do?

To get started, I would like real examples, except “I need 99 algorithms”

I gave you real example few messages ago and I’m talking about it right now. Tetrahedral mesh.

Similarly, I want to note that I already answered that that example does not interfere with using your own containers in the application.

Then what should I do? I’m not talking about my application.

Give a real example and I will tell you how to do it.

I gave it just few messages ago. Why do you keep ignoring it?
TetrahedralMesh. TetrahedralMesh. TetrahedralMesh. How many times I shall repeat it?

No need to be angry, just read what I wrote.

    VertexBuffer* buffer = originalModel->GetGeometry(0, 0)->GetVertexBuffer(0);
    const auto* vertexData = (const unsigned char*)buffer->Lock(0, buffer->GetVertexCount());

So, your actions with an array of vertices?

Ok, maybe I wasn’t clear enough. If you want to see the code, you can find a file with this name in *** repo.

Assuming that I don’t want to rewrite this class, how to contribute it to main Urho repo? Yes, I’m using standard containers there.

Please give a link to TetrahedralMesh (I don’t understand why such a class in the engine)

It’s quite annoying to search something from the phone…
https://github.com/***/***/blob/master/Source/Urho3D/Math/TetrahedralMesh.h

This class is mandatory for pre-computed global illumination and have wide range of applications, because it is essentially a mechanism for smooth sampling in 3D point cloud.

This class is mandatory for pre-computed global illumination and have wide range of applications, because it is essentially a mechanism for smooth sampling in 3D point cloud.

Well, there’s a complex example and I can’t get it right quickly. There, the number of vertices changes dynamically every frame or remains unchanged?

It is unchanged during the runtime, unless user triggers rebuild.
The main part of this class is pre-computing, tho.
I don’t see how it’s relevant.

ea::vector<unsigned> removedTetrahedrons;
ea::vector<unsigned> postponedVertices;
...
ea::swap(postponedVertices, verticesQueue);

Is this the example you are talking about?

The whole class is the example.
Example of code that cannot be easily merged into Urho master due to containers.

I thought the question is that Urho3D has bad containers, and not that it is not compatible with some other engine.

What line of code should I see to understand what is impossible to do with Urho3d containers

Urho has bad containers, therefore I don’t use them (just like you said) and therefore I cannot contribute anything container-related into Urho.

Who said “impossible”?
Nothing is impossible. You can write the same code without containers at all. The effort varies.
I don’t want to spend more than a few seconds on trivial algorithms, therefore I cannot write code for Urho containers.

Well, I can only speak with concrete examples. Which code snippet is more efficient on EASTL containers?

If you try to rewrite TetrahedralMesh for Urho containers without adding new code, you will find all such snippets. Or you can just look at standard algorithms used in this file (except swap and sort existing in Urho)

If you want one-line example:
https://github.com/***/***/blob/master/Source/Urho3D/Scene/Serializable.cpp#L684
This is EASTL-only feature. How would you rewrite it to Urho containers without adding extra allocation?

What does this container do? What is his particularity?

Vector without allocations.
Serialization shall not allocate, and I’m trying to move in this direction.
This is the first step.

In what sense without memory allocations? If you mean that the size is known in advance, you can specify capcity when creating a vector in Urho3D

In sense that it doesn’t call “new”.
Urho vector with non-zero size does it unconditionally.

Well, this is a compelling example. Let’s replace the Urho3D containers. True, we will lose 10% FPS, but serialization, which is performed occasionally, will be without memory allocation!

Do you have confirmed FPS loss caused by EASTL in real example?

I conducted tests and as I said, they are not perfect. Do you plan to change containers without any tests at all?

Nope. Tests are important.
And tests show that performance impact of containers is negligible, unless scene rendering is CPU-bound. And if scene rendering is CPU-bound, then containers impact is only a fraction of all suboptimal and weird things Urho does in its rendering. Just think about it: I barely have 60 FPS on my 6-cores i7 drawing just 63k boxes.

And tests show that performance impact of containers is negligible, unless scene rendering is CPU-bound.

That’s right, in the debug version of the engine I lose half the FPS due to the only thing - the sorting of batches. Batches are sorted every frame. Do you continue to argue that the performance of the containers is not important?

As far as I know, sorting of batches has nothing to do with container performance. Sorting algo and memory layout of batches determine performance of batch sorting.
Batch sorting performance is important and is completely unrelated to overall performance of container library. Batch sorting may be slower due to debug iterators, but it’s not the case neither for Urho nor of EASTL containers

OK, I think it’s worth continuing the conversation after real tests “engine with eastl” vs “engine with old containers”. At the moment I am busy with bindings and not interested in EASTL containers, so someone else should do this task (who could it be?)

And I’m not interested in EASTL in Urho too. Not now, at least.

As I said before, migration to standard containers is not about performance. Performance is worth checking, but performance loss may be overlooked if it happens only in synthetic benchmarks abusing the engine.

As you said, container performance is only important when rendering. Samples render objects every frame and from this point of view this test is not synthetic, but real.

And I’m not interested in EASTL in Urho too. Not now, at least.

From all the conversations, I realized that you plan to work on *** and sometimes throw a bone to a hungry dog. Or maybe you just want to facilitate the transfer of changes from Urho3D to ***. Because otherwise the fate of AtomicEngine will befall him.

Pretty much, yes. I contribute to Urho when it is possible, but I don’t want to fight with the engine every time I try to make something new.

The transfer from Urho to *** is already easy. EASTL (or just STL) in Urho would make backward transfer easy too.
Backward transfer from *** to Urho is low priority for me, that’s why I’m not actually pushing EASTL, only agitating.

Discourse told me 54 new posts! Wow. Well I guess that’s from chat style of conversation. Not criticizing, just surprised at big number. Feels like payback for “walls of text” write. So here’s another one.

C++ has the overriding issue of older code, C++11, C++14, C++17, and C++20. The better scripting language binding mechanisms, are requiring higher and higher versions, to achieve elegance and less maintenance work. For instance, sol2 3.0 uses C++17 features. There are 2 or 3 other possibly “good” C++ binding regimes available for Lua, I have not worked through their differences. But which version of C++, is clearly one of the tradeoffs. Meanwhile in Python land, pybind11 of course requires C++11. That’s pretty much bare minimum for “dragging people forwards”, going that route.

My research on EASTL vs. STL is that performance has to be checked. EASTL was for performance a long time ago. Various STL may be just fine now, particularly if dragging people forwards through C++## versions.

If dragging people through C+++## versions, then “niceness” arguments have to be checked too. You think EASTL is nice, what if we think STL in C++14 is nice? I don’t know or have an opinion on that at this time, but I do know, this is the question. Your EASTL may not be as future looking as you think.

The loops in your TetrahedralMesh code, look like the kinds of things I personally would want to do in something resembling ASM code. Because I actually think in ASM code, I have trouble committing to Python or Lua. I’m digging a bit through the Lua universe to understand how people try to use it for performance. It may not look like sanity from my ASM perspective. I don’t know if your container approach looks like sanity either. I understand “niceness” and “get it working” arguments, but it’s a C++ perspective and I don’t like C++. If I had a better language to offer you, I’d try to convince you to write stuff in that, and forget about this EASTL or STL container stuff.

I am going to look at what Jonathan Blow thinks of LLVM. I believe he is writing Jai on top of that. Using LLVM “as portable assembly” is something other people have done. However LLVM is a big heavy external dependency, a pile of code. Historically I did not want to get into it, because I cannot personally maintain LLVM or take it into the future. But if Blow actually ships Jai in the near future, then getting into LLVM makes more sense to me. I wouldn’t be “alone” in dealing with it for 3d graphics and game performance issues.

The Lua way of doing this is much more sane by comparison. You can wrap your head around the standard Lua implementation. It’s designed to be good at that. But if the standard Lua 5.4 doesn’t actually do what I want, well it’s a problem.

I suspect that LuaJIT probably does more like what I want, haven’t finished investigating. But it’s a development dead end. No leadership, headless. Clearly, they can’t stay abreast of official Lua, which harms the ecosystem.

PyPy does actually stay abreast of Python, it’s fine that way. But you have to deal with the GIL, which could be suicide for all this low level manipulation stuff. Having to even think about the GIL, definitely makes me itch. I really don’t want to itch.

I don’t think these people write their languages from a 3D programmer’s perspective. Someone who has to manipulate a lot of vector math in arrays.

I revisited Julia, but apparently their JIT is slow to get fired up. They don’t have any kind of Ahead Of Time compilation, and do not seem to be oriented towards applications development. Rather, scientific computation batch processing. Software amateurs fiddling around to get their numeric results. Nobody developing games touches it. Get the idea that devs might like their language to be more general purpose and not sitting in its niche, but they aren’t meaningfully executing on that. Wait more years, see if they moved forwards any, seems to me.

On the other hand, they had 2 C++ binding mechanisms available. I started to yawn. Might double check my work sometime.

I would be happy if we could move Urho to standard library.
Because that’s why standard libraries are created, right? To be used.

My point is that migration to pure STL is significantly harder than migration to augmented EASTL.
And the lower version of STL you aim, the less benefits will be gained from such migration.

E.g. I like EASTL for spans and string views, but they were standartized only in C++20.
I think that only C++17 or C++20 allow user to write their own stack-allocating containers, while EASTL has them already implemented.

Well if you insist per previous arguments, that EASTL inclusion has to break existing UrhoContainer, then it isn’t any more of a tempting offer than C++## STL stuff. From the user’s perspective, breaking stuff is breaking stuff. It’s churn.

My hope would be that EASTL or one of the C++## STL does not have to break UrhoContainer, that they can live side by side, even if there is some performance loss for doing so. You say you looked at it, didn’t find any answer, and just want to make some big break.

I haven’t looked at it. I assume from your failed efforts, “it is a hairy problem”. Touches all kinds of things. Will probably not look at it for some time. I wrap my head around Urho3D and scripting language “theory of operation”. Seems to be very basic and fundamental question, how people choose to express their vector math in code.

I started looking at C++## migration issues “in general” in industry. Not great guidance TBH. Having new language stuff every 3 years nowadays, doesn’t actually strike me as being a feature. Seems like churn. The best thing I’ve found so far is Hitler on C++17:

don’t be scared… they will fix variadic…

meh… who did this is fuc*/&%% genius

go getting a Ruby book

It brings all new meaning to “Code Nazi”.