Rebirth getting weird control types and changing HOG files
#11
That is the exception log, not the memory dump. The dump should be generated automatically when the log is written, and the message box telling you about the log will also report the path to the memory dump. However, dump support is optional. Some builds lack it. If your build lacks it, it will not be generated nor mentioned in the message box. I know AFP produced at least one build without dump support, but I thought he subsequently fixed that and posted builds with dump support.

Regardless, that log confirms that the client mismatched on the level integrity checksum. Now we need to know who was right: the host or the client. Since you say that the host needs to restart to clear this, my guess is that the host is wrong and the client is right. If the client was wrong, the forced restart associated with that exception would have cleared the issue. Is this problem reproducible with the latest code? Is it reproducible in 0.58.1? If you swap host and guest, does the error stay with the computer or follow to the new host?
Reply
#12
Gonna be away from the computer for about a week; I'll test as soon as I can.
Reply
#13
Remembered this was a thing after it happened again in a different mission. It is a host-independent error that does not occur in 0.58.1; I have not tested in newer versions of the game.

I have however traced the issue to dying while trying to escape. Apparently when the guest dies, only he gets the error; when the host dies both get the error. I have not tested when both people fail to escape.
Reply
#14
There is a known issue in the 0.60 beta2 associated with dying while escaping. It was previously reported by Ninjared in CTD if not exit mine in multiplayer. I fixed that in Hide game window when advancing a dead player to the next level (December 2018). This commit was months after the beta you are using, which is why I always encourage reporters to test the latest code. Please test the latest code and report whether you still encounter a crash.
Reply
#15
Similar issue here.
D2X-rebirth v0.61
Complied on Linux for Linux play.

The situation revolving around the 14 files needed from the official descent 2 for use with rebirth.
(On Linux) playing mutiplayer on local network; everything going smoothly untill the joined player dies. When the joined player dies they can not rejoin, with error message "your level file does not match the other players. Cannot join game."

To resolve this, both the host and the client need to exit game and quit. Then delete the "/home/.d2x-rebirth/Data" folder contents, and recopy the files (14total) from the original game. Then restart the mutiplayer. Then player can join again.

I can't see that playing the game should in any way alter these files beyond the record of "last accessed"

 I did compile Linux versions of dxx-rebirth for both computers. Mine an arch Linux (Manjaro) and my sons Ubuntu 19.
So wine or the windows exe is in no way a contributing factor.

We played all day Sunday. Through 11 levels.  I hosted the games because I'm dad, and the household champion. Hahaha
Anyway, he died on most of those levels. (But not every level) at least once. 
After the first time it happened on Saturday night (test night) we made a backup folder on both machines with these original files from the purchased game.

So on sunday, even tho it was a hassle, we had a routine.
He would die. And try to rejoin only to get the error. Then I would save the mutiplayer game. We both shut down the game. Delete the /data/ folder and contents. Then copy our back up files into place and restart. He'd join with out issue and I'd load the saved game.

But like I said. The original purchased descent files are only being read and loaded. There is no writing to those files....as far as I can imagine. Right?

I'm just not sure.... As far as size or md5 hashing is concerned there appears no change... The only change I can see on those files is "last accessed" flags. But surely that would have no effect. Because if decent is looking at those details, nobody anywhere would be able to multiplay games... Not even online. Like everybody's "last accessed" flag is more likely than not going to be different. 
For the first time in my life I wish I had two windows machines to see if it also occurs on the windows version... To eliminate "it's a Linux issue." 
 But I'm almost certian playing the game is not writing changes to the original level files.....

@KP tell what you need me to do. I want to contribute. I'm not a game developer but am an Android Developer (recognized on XDA) and totally not afraid to break things or get my hands dirty. If I can get my head around what's happening here I will submit an official issue on the git. Just not sure where to look or what to look for just yet. (Just two weeks into discovery of dxx-rebirth)

Thanks for your time and all your hard work. My son and I really appreciate what you do!
Reply
#16
(12-26-2018, 03:30 AM)Kp Wrote: The game checksums select segment data and sends the checksum.  The full calculation is in netmisc_calc_checksum in similar/main/gameseq.cpp.

It would be a bit surprising that a wild write could manage to set a valid value, but it's possible.  The levels aren't kept loaded when you're not playing them, so an early wild write shouldn't be able to corrupt a later level. [color:red] The corruption would need to happen after the level loads, but before you can play it.

Can you provide a memory dump when the game reports the slew error?

Sounds eeriely familiar. The guest player dies. Is kicked out to menu (I assume this is the proper behaviour) when he tried to rejoin, he's greeted with the error message; "your level file does not match the other players. Cannot join game" but on the host I see message
player xxx has joined game
followed by
player xxx has left game

Almost with in 1 second of each other.

How can I retrieve this memory dump?
Reply
#17
Quick note - if you're doing personal compiles, please include the version string shown when the game starts. This is visible on the main menu page (where it is an image, so cannot be copied, but is easy to visually inspect to see if it is what you expect), is printed to stdout near the top of the output (if you run from a terminal), and is in your gamelog.txt if you want to get it from there (always available, regardless of platform). As an example string, which won't match yours because I had some unpublished changes when I got this one, I recently got D1X-Rebirth v0.61 0.60.0-beta2-413-g7bd861e37493*.

On to your problem:
Getting killed should not dump either player from the game. If it does, that's a bug in its own right, even if the killed player could immediately rejoin.
Getting that error on rejoin is obviously a bug, and it's weird that I now have multiple people reporting that they have to delete the data directory. As you say, that shouldn't be required at all. The game doesn't open the data files for write, and in fact should run correctly even when the data files are read-only. For me, the Descent 2 data files are all installed under /usr/share/games/d2x and are root:root 644, and the game has no problem with me playing as my user, not as root.

OldSaltyGamer Wrote:For the first time in my life I wish I had two windows machines to see if it also occurs on the windows version... To eliminate "it's a Linux issue."
I use Linux as my main environment, and all my testing is done on Linux too. Even if it is "just a Linux issue", it would matter quite a bit to me. Smile

I just set up a local multiplayer game in Descent 2, on the first level, as a cooperative game. On the guest player, I killed myself with a point-blank concussion missile to a wall. I was not kicked from the game at death or when I respawned.

For the memory dump, the quick and dirty solution would be to kill both games with a signal that causes a core dump, such as SIGQUIT. The cleaner solution would be to attach gdb and use generate-core-file to write a core file without killing the game. However, the core file will only be useful to me if I have good symbolic data for the associated program. You may not have built with symbols at all, and if you did, they will be specific to the build you created. It's very unlikely that I could use symbols from my builds here to read your core file. If we were chasing a crash bug, I could just tell you to post-process the core file with gdb to extract the backtraces and variable values I want. Since we're chasing a data mismatch bug, I need to know what data mismatches, which potentially means inspecting all of it. One way to handle this, at the expense of a bit of extra output, would be for you to rebuild both games with some targeted debug logging to print out the checksum data as it goes. The below patch will write that to your gamelog. After the error, save both game logs and compare them. The output generated by the patch in gameseq.cpp should be identical if everything works correctly. I expect it will not be, given the error you reported. I expect only one person will have the output from net_udp.cpp. That output will still be useful, because it will confirm whether the checksum computed by the peer is the one received by the writer.

As an aside, you may want to move this to Github sooner rather than later. According to the front page, zico is looking to close the forums at some indeterminate date. I'm not moving to Discord, but will continue to provide support through Github.
Code:
diff --git a/similar/main/gameseq.cpp b/similar/main/gameseq.cpp
index 123a676bb..3f54fd0ee 100644
--- a/similar/main/gameseq.cpp
+++ b/similar/main/gameseq.cpp
@@ -861,9 +861,18 @@ static const d_fname &get_level_file(int level_num)
}

// routine to calculate the checksum of the segments.
-static void do_checksum_calc(const uint8_t *b, int len, unsigned int *s1, unsigned int *s2)
+static void do_checksum_calc(const uint8_t *b, int len, unsigned int *s1, unsigned int *s2, const char *file, unsigned line)
+#define do_checksum_calc(B,L,S1,S2)    do_checksum_calc(B,L,S1,S2,__FILE__,__LINE__)
{
-
+    con_printf(CON_URGENT, "%s:%u: %s: b=%p len=%i sum1=%08x sum2=%08x [%08x]", file, line, __func__, b, len, *s1, *s2,
+               len == 1
+               ? static_cast<unsigned>(*b)
+               : len == 2
+               ? static_cast<unsigned>(*reinterpret_cast<const uint16_t *>(b))
+               : len == 4
+               ? static_cast<unsigned>(*reinterpret_cast<const uint32_t *>(b))
+               : 0xcdcdcdcd
+               );
    while(len--) {
        *s1 += *b++;
        if (*s1 >= 255) *s1 -= 255;
@@ -879,11 +888,14 @@ static ushort netmisc_calc_checksum()
    int t;

    sum1 = sum2 = 0;
-    range_for (auto &&segp, vcsegptr)
+    con_printf(CON_URGENT, "%s:%u: %s: mission=\"%s\" level=%i", __FILE__, __LINE__, __func__, Current_mission->path.c_str(), Current_level_num);
+    range_for (auto &&segp, vcsegptridx)
    {
        auto &i = *segp;
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (auto &&e, enumerate(i.shared_segment::sides))    // d_zip
        {
+            con_printf(CON_URGENT, "%s:%u: %s: seg[%hu].side[%lu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), e.idx, sum1, sum2);
            auto &j = e.value;
            do_checksum_calc(reinterpret_cast<const uint8_t *>(&(j.get_type())), 1, &sum1, &sum2);
            s = INTEL_SHORT(j.wall_num);
@@ -912,16 +924,19 @@ static ushort netmisc_calc_checksum()
                do_checksum_calc(reinterpret_cast<uint8_t *>(&t), 4, &sum1, &sum2);
            }
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (auto &j, i.children)
        {
            s = INTEL_SHORT(j);
            do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (const uint16_t j, i.verts)
        {
            s = INTEL_SHORT(j);
            do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        s = INTEL_SHORT(i.objects);
        do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
#if defined(DXX_BUILD_DESCENT_I)
@@ -931,7 +946,9 @@ static ushort netmisc_calc_checksum()
        do_checksum_calc(reinterpret_cast<uint8_t *>(&t), 4, &sum1, &sum2);
#endif
        do_checksum_calc(&i.station_idx, 1, &sum1, &sum2);
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
    }
+    con_printf(CON_URGENT, "%s:%u: %s: done: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, sum1, sum2);
    sum2 %= 255;
    return ((sum1<<8)+ sum2);
}
diff --git a/similar/main/net_udp.cpp b/similar/main/net_udp.cpp
index cc4bf4922..174db8f27 100644
--- a/similar/main/net_udp.cpp
+++ b/similar/main/net_udp.cpp
@@ -4096,6 +4096,7 @@ void net_udp_read_sync_packet(const uint8_t * data, uint_fast32_t data_len, cons
    GameUniqueState.Difficulty_level = Netgame.difficulty;
    Network_status = Netgame.game_status;

+    con_printf(CON_URGENT, "%s:%u: %s: Netgame.segments_checksum=%#hx my_segments_checksum=%#hx", __FILE__, __LINE__, __func__, Netgame.segments_checksum, my_segments_checksum);
    if (Netgame.segments_checksum != my_segments_checksum)
    {
        Network_status = NETSTAT_MENU;
The forum is infamous for mangling patch files. If you copy it from the thread view, it won't work. Open a reply to my post, then copy from the text input box showing my quoted message. In that box, tabs will be tabs and you should be able to get a good patch file.

[Edit: improved patch output.]
Reply
#18
Quote:The cleaner solution would be to attach
Quote:gdb and use generate-core-file
Quote: to write a core file without killing the game.  However, the core file will only be useful to me if I have good symbolic data for the associated program.  You may not have built with symbols at all, and if you did, they will be specific to the build you created.  It's very unlikely that I could use symbols from my builds here to read your core file.  If we were chasing a crash bug, I could just tell you to post-process the core file with gdb to extract the backtraces and variable values I want.  Since we're chasing a data mismatch bug, I need to know what data mismatches, which potentially means inspecting all of it.  One way to handle this, at the expense of a bit of extra output, would be for you to rebuild both games with some targeted debug logging to print out the checksum data as it goes.  The below patch will write that to your gamelog.  After the error, save both game logs and compare them.  The output generated by the patch in gameseq.cpp should be identical if everything works correctly.  I expect it will not be, given the error you reported.  I expect only one person will have the output from net_udp.cpp.  That output will still be useful, because it will confirm whether the checksum computed by the peer is the one received by the writer.
As an aside, you may want to move this to Github sooner rather than later.  According to the front page, zico is looking to close the forums at some indeterminate date.  I'm not moving to Discord, but will continue to provide support through Github.
Code:
diff --git a/similar/main/gameseq.cpp b/similar/main/gameseq.cpp
index 123a676bb..1d1d2867a 100644
--- a/similar/main/gameseq.cpp
+++ b/similar/main/gameseq.cpp
@@ -861,9 +861,16 @@ static const d_fname &get_level_file(int level_num)
}

// routine to calculate the checksum of the segments.
-static void do_checksum_calc(const uint8_t *b, int len, unsigned int *s1, unsigned int *s2)
+static void do_checksum_calc(const uint8_t *b, int len, unsigned int *s1, unsigned int *s2, const char *file, unsigned line)
+#define do_checksum_calc(B,L,S1,S2)    do_checksum_calc(B,L,S1,S2,__FILE__,__LINE__)
{
-
+    con_printf(CON_URGENT, "%s:%u: %s: b=%p len=%i sum1=%08x sum2=%08x [%08x]", file, line, __func__, b, len, *s1, *s2,
+               len == 2
+               ? static_cast<unsigned>(*reinterpret_cast<const uint16_t *>(b))
+               : len == 4
+               ? static_cast<unsigned>(*reinterpret_cast<const uint32_t *>(b))
+               : 0xcdcdcdcd
+               );
    while(len--) {
        *s1 += *b++;
        if (*s1 >= 255) *s1 -= 255;
@@ -879,11 +886,14 @@ static ushort netmisc_calc_checksum()
    int t;

    sum1 = sum2 = 0;
-    range_for (auto &&segp, vcsegptr)
+    con_printf(CON_URGENT, "%s:%u: %s: mission=\"%s\" level=%i", __FILE__, __LINE__, __func__, Current_mission->path.c_str(), Current_level_num);
+    range_for (auto &&segp, vcsegptridx)
    {
        auto &i = *segp;
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (auto &&e, enumerate(i.shared_segment::sides))    // d_zip
        {
+            con_printf(CON_URGENT, "%s:%u: %s: seg[%hu].side[%lu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), e.idx, sum1, sum2);
            auto &j = e.value;
            do_checksum_calc(reinterpret_cast<const uint8_t *>(&(j.get_type())), 1, &sum1, &sum2);
            s = INTEL_SHORT(j.wall_num);
@@ -912,16 +922,19 @@ static ushort netmisc_calc_checksum()
                do_checksum_calc(reinterpret_cast<uint8_t *>(&t), 4, &sum1, &sum2);
            }
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (auto &j, i.children)
        {
            s = INTEL_SHORT(j);
            do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        range_for (const uint16_t j, i.verts)
        {
            s = INTEL_SHORT(j);
            do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
        }
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
        s = INTEL_SHORT(i.objects);
        do_checksum_calc(reinterpret_cast<uint8_t *>(&s), 2, &sum1, &sum2);
#if defined(DXX_BUILD_DESCENT_I)
@@ -931,7 +944,9 @@ static ushort netmisc_calc_checksum()
        do_checksum_calc(reinterpret_cast<uint8_t *>(&t), 4, &sum1, &sum2);
#endif
        do_checksum_calc(&i.station_idx, 1, &sum1, &sum2);
+        con_printf(CON_URGENT, "%s:%u: %s: seg[%hu]: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, segp.get_unchecked_index(), sum1, sum2);
    }
+    con_printf(CON_URGENT, "%s:%u: %s: done: sum1=%08x sum2=%08x", __FILE__, __LINE__, __func__, sum1, sum2);
    sum2 %= 255;
    return ((sum1<<8)+ sum2);
}
diff --git a/similar/main/net_udp.cpp b/similar/main/net_udp.cpp
index 78ef38c0b..1ab2f5f24 100644
--- a/similar/main/net_udp.cpp
+++ b/similar/main/net_udp.cpp
@@ -4096,6 +4096,7 @@ void net_udp_read_sync_packet(const uint8_t * data, uint_fast32_t data_len, cons
    GameUniqueState.Difficulty_level = Netgame.difficulty;
    Network_status = Netgame.game_status;

+    con_printf(CON_URGENT, "%s:%u: %s: Netgame.segments_checksum=%hu my_segments_checksum=%hu", __FILE__, __LINE__, __func__, Netgame.segments_checksum, my_segments_checksum);
    if (Netgame.segments_checksum != my_segments_checksum)
    {
        Network_status = NETSTAT_MENU;
The forum is infamous for mangling patch files.  If you copy it from the thread view, it won't work.  Open a reply to my post, then copy from the text input box showing my quoted message.  In that box, tabs will be tabs and you should be able to get a good patch file.

I'll get this all done over the course of the next 24 hours.

1. Apply the patch
2. Rebuild for both machines (historically I just have typed in "scons" and let her build. If you want symbols I dare request what arguments to append to the command.)
3. Play game with son and after error has occurred;
4. Set aside and review gamelog and gdb core file.
5. Issue a bug report on the git of my findings and relivant info.
6. Post up the gamelog and the gdb core file.

Thanks for your support!
Reply
#19
Btw. I just mean to say about the whole wishing I had two windows machines, is I have 5 Linux computers in my home. Lol
I didn't mean to leave you with the impression I thought you didn't care about Linux issues. 
My bad.
I just can't help but wonder if my son and I were both on windows would the bug still be present. But I'll never be able to know that.

Also. I'm building v0.61
Reply
#20
similar/main/gameseq.cpp:956: netmisc_calc_checksum: seg[500]: sum1=000000f3 sum2=01eae351
similar/main/gameseq.cpp:958: netmisc_calc_checksum: done: sum1=000000f3 sum2=01eae351
fluidsynth: warning: Failed to pin the sample data to RAM; swapping is possible.
Found 4 cooperative starts in mission "d2" level 10.

** (process:5401): CRITICAL **: 18:21:19.324: fluid_synth_cc: assertion 'val >= 0 && val <= 127' failed

** (process:5401): CRITICAL **: 18:21:19.324: fluid_synth_cc: assertion 'val >= 0 && val <= 127' failed
similar/main/net_udp.cpp:4096: net_udp_read_sync_packet: Netgame.segments_checksum=36661 my_segments_checksum=62241
terminate called after throwing an instance of 'multi::level_checksum_mismatch'
what(): level checksum mismatch
Aborted (core dumped)

http://theandroidcollective.org/dl/D2X-r...162019.zip
Reply


Forum Jump:


Users browsing this thread: 4 Guest(s)